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EXECUTIVE  SUMMARY 


U.S.  Army  contingency  contracting  officers  (CCOs)  play  a  critical  role  in  nearly  all 
overseas  military  operations  by  providing  supplies  and  services  to  the  warfighter  in  an 
efficient,  timely,  and  ethical  manner.  Despite  this  critical  role,  the  Anny  has  not 
integrated  its  CCOs  or  the  procurement  process  in  a  holistic  approach  to  its 
expeditionary,  warfighting  mindset.  Shortfalls  in  the  number  of  U.S.  Army  CCO 
personnel  and  the  lack  of  a  standardized  training  process  have  negatively  affected  the 
Army’s  expeditionary  procurement  capabilities. 

The  U.S.  Army’s  Expeditionary  Contracting  Command  (ECC)  was  established  in 
January  2008  in  direct  response  to  the  Gansler  Commission’s  recommendations  to 
prepare,  train,  and  provide  skilled  contracting  personnel  in  support  of  U.S.  Army 
contingency  operations.  This  new  command,  and  other  resultant  changes  of  the  Gansler 
Commission’s  recommendations,  resulted  in  a  planned  net  increase  of  over  400 
uniformed  and  800  civilian  contracting  officer  positions. 

The  ECC  developed  a  standardized  set  of  36  tasks  to  properly  train  CCOs  prior  to 
deployment.  These  36  tasks,  published  in  the  ECC’s  Soldier’s  Manual  of  Common  Tasks 
(SMCT)  MOS  5 1C  Acquisition,  Logistics,  and  Technology,  represent  the  body  of 
knowledge  that  a  5 1C  CCO  must  know  to  be  successful  during  contingency  operations. 

The  establishment  and  standardization  of  training  tasks  by  the  ECC  for  CCOs  in 
the  MOS  5 1C  SMCT,  in  accordance  with  Army  training  doctrine,  has  helped  subordinate 
units  and  individual  CCOs  within  the  ECC  to  focus  their  training  efforts  prior  to 
deployment.  This  body  of  knowledge  is  a  significant  step  in  standardizing  the  pre¬ 
deployment  training  for  CCOs  and  preparing  them  for  their  mission  once  deployed.  To 
supplement  this  training,  an  appropriate  performance-assessment  tool  is  needed  to 
measure  performance  of  these  36  tasks.  Currently,  the  ECC  lacks  a  standardized  method 
of  measuring  individual  skill  and  task  proficiency  in  its  CCOs. 
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A  written  proficiency-assessment  test  will  allow  leaders  in  the  ECC  to  measure  a 
CCO’s  knowledge,  skills,  and  capabilities.  A  written  PAT  will  enable  leaders  to  measure 
individual  task  proficiency  against  established  standards,  determine  each  CCO’s 
preparedness  to  conduct  duties  during  deployment,  make  appropriate  CCO  assignments, 
and  develop  individualized  training  programs  for  CCOs. 

This  research  developed  the  questions  for  a  written  proficiency  test  based  on  the 
36  tasks  listed  in  the  SMCT.  The  research  team  developed  the  scenarios  and  questions  to 
assess  proficiency  in  accordance  with  conditions,  standards,  and  performance  steps  for 
each  of  the  36  tasks  in  the  SMCT.  Questions  were  structured  to  assess  Bloom’s  cognitive 
domains  of  remembering,  understanding,  and,  to  a  limited  extent,  application  of  tasks. 
The  research  team  vetted  each  of  the  scenarios  and  accompanying  questions  through 
experienced  CCO  focus  groups  within  the  ECC.  Upon  completion  of  the  test,  the 
research  team  provided  a  written  Contracting  Officer  Proficiency-Assessment  Test, 
encompassing  over  1,100  questions  covering  each  of  the  36  tasks  listed  in  the  SMCT ,  to 
the  Expeditionary  Contracting  Command. 
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I.  INTRODUCTION 


A.  BACKGROUND 

U.S.  Army  contingency  contracting  officers  (CCOs)  play  a  critical  role  in  nearly 
all  overseas  military  operations  by  providing  supplies  and  services  to  the  warfighter  in  an 
efficient,  timely,  and  ethical  manner.  Despite  this  critical  role,  the  Anny  has  not 
integrated  its  CCOs  or  the  procurement  process  in  a  holistic  approach  to  its 
expeditionary,  warfighting  mindset.  Shortfalls  in  the  number  of  U.S.  Army  CCO 
personnel  and  the  lack  of  a  standardized  training  process  have  negatively  affected  the 
Army’s  expeditionary  procurement  capabilities  (Gansler,  2007). 

The  21st  century  has  brought  about  significant  changes  in  the  manner  in  which 
U.S.  Forces  conduct  operations  overseas.  The  post-Cold  War  reduction  in  forces  saw  a 
decrease  in  acquisition  workforce  professionals  throughout  the  Army.  At  the  same  time, 
the  United  States  military,  and  in  particular  the  Anny,  significantly  increased  the 
outsourcing  of  many  jobs  that  were  previously  perfonned  by  uniformed  Service  members 
(Gansler,  2007).  This  decrease  in  contracting  professionals  and  increase  in  contracting 
requirements  led  to  many  deficiencies  highlighted  in  the  Gansler  Commission  report 
(2007). 

The  U.S.  Army’s  Expeditionary  Contracting  Command  (ECC)  was  established  in 
January  2008  in  direct  response  to  the  Gansler  Commission’s  recommendations  to 
prepare,  train,  and  provide  skilled  contracting  personnel  in  support  of  U.S.  Army 
contingency  operations  (Gansler,  2007).  This  new  command,  and  other  resultant  changes 
of  the  Gansler  Commission’s  recommendations,  resulted  in  a  planned  net  increase  of  over 
400  uniformed  and  800  civilian  contracting  officer  positions  (DoA,  2009a).  Most  of  the 
new  uniformed  positions  within  the  ECC  are  coded  “5 1C,”  the  U.S.  Army’s  Military 
Occupational  Specialty  (MOS)  code  for  contracting  officer.  The  ECC  developed  a 
standardized  set  of  tasks  to  properly  train  CCOs  prior  to  deployment.  These  36  tasks, 
published  in  the  ECC’s  Soldier’s  Manual  of  Common  Tasks  (SMCT)  MOS  5 1C 
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Acquisition,  Logistics,  and  Technology,  represent  the  body  of  knowledge  that  a  5 1C  CCO 
must  know  to  be  successful  during  contingency  operations  (DoA,  2009a). 

This  research  builds  upon  the  36  tasks  listed  in  MOS  5 1C  SMCT  by  building  a 
proficiency-assessment  test  (PAT)  for  the  SMCT  tasks.  The  PAT  developed  throughout 
this  research  is  nested  with  Army  training  doctrine  prescribed  in  U.S.  Army  Field  Manual 
(FM)  7-0  and  provides  an  assessment  and  evaluation  tool  for  individual  task  proficiency 
in  CCOs.  This  assessment  “provide[s]  the  link  between  the  unit’s  [or  individual’s] 
performance  and  the  [established]  Army  standard,”  as  prescribed  in  the  SMCT  (DoA, 
2002). 

B.  PROBLEM  STATEMENT 

The  establishment  and  standardization  of  training  tasks  by  the  ECC  for  CCOs  in 
the  MOS  5 1C  SMCT,  in  accordance  with  Army  training  doctrine,  has  helped  subordinate 
units  and  individual  CCOs  within  the  ECC  to  focus  their  training  efforts  prior  to 
deployment.  This  body  of  knowledge  is  a  significant  step  in  standardizing  the  pre¬ 
deployment  training  for  CCOs  and  preparing  them  for  their  mission  once  deployed.  To 
supplement  this  training,  an  appropriate  performance-assessment  tool  is  needed  to 
measure  performance  of  these  36  tasks.  Currently,  the  ECC  lacks  a  standardized  method 
of  measuring  individual  skill  and  task  proficiency  in  its  CCOs.  The  ECC  detennines 
CCO  duty  assignments  and  warrant  levels  without  a  quantifiable  measure  of  5 1C 
individual  task  proficiency.  A  written  proficiency-assessment  test  will  allow  leaders  in 
the  ECC  to  measure  a  CCO’s  knowledge,  skills,  and  capabilities.  A  written  PAT  will 
enable  leaders  to  measure  individual  task  proficiency  against  established  standards, 
determine  each  CCO’s  preparedness  to  conduct  duties  during  deployment,  make 
appropriate  CCO  assignments,  and  develop  individualized  training  programs  for  CCOs. 

C.  RESEARCH  OBJECTIVES 

The  objective  of  this  research  is  to  develop  a  written  proficiency  test  for  the  36 

tasks  listed  in  the  SMCT.  The  research  team  developed  the  scenarios  and  questions  to 

assess  proficiency  in  accordance  with  conditions,  standards,  and  performance  steps  for 

each  of  the  36  tasks  in  the  SMCT.  Questions  were  structured  to  assess  Bloom’s  cognitive 
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domains  of  remembering,  understanding,  and,  to  a  limited  extent,  application  of  tasks 
(Bloom,  1956).  The  research  team  vetted  each  of  the  scenarios  and  accompanying 
questions  through  experienced  CCO  focus  groups  within  the  ECC.  Upon  completion  of 
the  test,  we  provided  a  written  Contracting  Officer  Proficiency-Assessment  Test, 
covering  each  of  the  36  tasks  listed  in  the  SMCT,  to  the  Expeditionary  Contracting 
Command.  This  final  document  included  written  documentation  of  scenarios,  questions, 
and  answers  developed  by  us. 

To  meet  this  objective,  this  research  answered  the  following  primary  research 
question: 

What  is  the  most  effective  and  efficient  method  of  measuring  Individual  CCO 
task  proficiency  in  the  36  tasks  listed  in  the  SMCT1 
Secondary  research  questions  were  as  follows: 

•  How  can  a  proficiency-assessment  test  be  developed? 

•  What  type(s)  of  testing  is  best  suited  to  evaluate  the  cognitive  domains  of 
remembering,  understanding,  and,  to  a  limited  extent,  application  of 
knowledge? 

•  What  type(s)  of  testing  is  most  efficient — in  terms  of  requiring  the  least 
amount  of  resources,  such  as  overhead  and  setup — for  the  tester  and  test 
taker? 

This  research  was  completed  when  scenario(s)  and  accompanying  questions  for 
each  of  the  36  tasks  had  been  developed,  the  scenarios  and  questions  were  vetted  through 
focus  groups  of  the  ECC,  and  a  final  written  version  of  the  PAT  was  submitted  and 
received  by  the  ECC.  Additionally,  the  research  team  provided  the  ECC  with 
recommendations  as  to  the  most  effective  format  in  which  to  issue  the  test. 
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D.  SCOPE  AND  DELIVERABLES 

The  scope  of  this  research  project  was  limited  to  developing  a  test  to  assess 
proficiency  in  the  36  tasks  listed  in  the  SMCT.  The  specific  deliverables  at  the 
conclusion  of  this  research  project  were  specifically  designed  to  satisfy  the  research 
project  objectives.  The  research  team  accomplished  the  following: 

1 .  Developed  a  written  proficiency-assessment  test  (PAT)  that  measures  proficiency 
in  each  of  the  36  tasks  listed  in  the  SMCT.  The  PAT  included  the  following: 

a.  Written  scenarios  for  each  of  the  36  tasks. 

b.  Questions  for  each  of  the  scenarios  that  measure  the  cognitive  domains  of 
remembering,  understanding,  and,  to  a  limited  extent,  application  of 
knowledge  of  the  36  tasks. 

2.  Developed  the  scenarios  and  accompanying  questions  to  assess  proficiency  in 
accordance  with  the  conditions,  standards,  and  perfonnance  steps  listed  in  the 
SMCT. 

3.  Vetted  each  scenario  and  the  accompanying  questions  through  focus  groups 
comprised  of  experienced  CCOs  and  members  of  the  ECC  to  ensure  the  scenario 
and  questions  satisfy  the  conditions,  standards,  and  perfonnance  steps  listed  in  the 
SMCT. 

4.  Built  the  PAT  in  a  manner  that  allows  it  to  be  converted  into  a  computerized 
assessment  test. 

5.  Provided  a  written  version  of  scenarios,  questions,  and  answers  to  the  ECC  upon 
completion. 

The  ECC  has  established  the  36  tasks  in  the  SMCT  as  the  body  of  knowledge  that 
a  CCO  must  know  to  be  successful.  The  research  team  accepted  these  tasks  and  focused 
on  developing  the  test  to  satisfy  proficiency  in  that  body  of  knowledge.  Analysis  of  the 
relevance  or  validity  of  the  36  tasks  as  they  relate  to  current,  changing,  or  emerging 
trends  in  CCO  tasks  were  not  within  the  scope  of  this  project.  As  such,  this  research  did 
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not  analyze  the  validity  of  tasks  listed  in  the  SMCT.  However,  validation  of  tasks  could 
be  a  subject  area  of  further  research  interest  by  follow-on  researchers. 

E.  POTENTIAL  BENEFITS 

A  written  proficiency-assessment  test  will  provide  essential  feedback  to  Army 
contracting  leaders  by  enabling  them  to  measure  individual  task  proficiency  against 
established  standards.  Leaders  could  administer  the  PAT  to  newly  arrived  CCOs  to 
gauge  what  tasks  they  are  proficient  in  and  what  tasks  they  need  work  in.  This 
knowledge  will  allow  CCOs  and  their  leaders  to  develop  specifically  tailored  and 
individualized  training  programs  to  address  weaknesses  in  the  CCOs  knowledge  base,  as 
demonstrated  in  the  PAT  results.  Follow-on  PATs  can  help  the  leader  and  CCO  see 
progress  in  the  individual  training  program.  Using  the  PAT  as  the  training  feedback 
mechanism  will  allow  leaders  and  individuals  to  detennine  if  their  training  focus  and 
methodology  is  accurate  or  should  be  adjusted.  Moreover,  the  PAT  will  serve  as  valuable 
feedback  to  a  commander  on  his  unit’s  training  program. 

Leaders  in  the  ECC  will  be  able  to  measure  a  CCO’s  knowledge,  skills,  and 
capabilities  and  detennine  a  CCO’s  preparedness  to  conduct  duties  during  deployment. 
The  PAT  can  serve  as  a  capstone  certification  of  an  individual’s  deployment  readiness  at 
the  conclusion  of  an  extensive  training  program  and  prior  to  a  scheduled  deployment. 
The  successful  passing  of  the  PAT  will  give  the  individual  CCO  confidence  in  his 
abilities  to  perform  his  mission.  Moreover,  it  will  serve  as  a  validation  to  the  individual’s 
leaders  that  the  CCO  is  ready  to  accomplish  his  mission. 

A  PAT  may  help  ECC  leaders  assign  CCOs  to  positions  that  match  the  CCO’s 
experience  and  skill  qualifications.  The  ECC  has  a  variety  of  positions  available  for 
assignment,  some  coded  for  more  experienced  CCOs  and  some  for  entry-level  CCOs. 
Two  classic  examples  of  positions  are  the  CCO  team  member  and  CCO  team  leader.  The 
first-line  supervisors  make  detailed  assignment  adjustments  at  the  unit  level  once  a  CCO 
has  arrived  and  has  been  assessed  over  time.  A  PAT  would  allow  ECC  leaders  and  first 
line  supervisors  to  distinguish  between  newly  arrived  or  inexperienced  CCOs  and  ones 
with  more  experience. 


5 


F. 


LIMITATIONS 


The  PAT,  as  developed,  has  several  limitations.  The  limitations  of  the  final 
version  of  the  PAT  and  its  accompanying  answers  include  susceptibility  to  compromise 
by  would-be  test  takers,  a  limited  ability  to  adapt  to  emerging  trends  in  contingency 
contracting,  inability  to  accurately  reproduce  the  interface  and  interaction  required  of 
CCOs,  and  inability  to  measure  experience  accurately.  Leaders  within  the  ECC  can  issue 
the  PAT  with  effective  results  if  they  understand  these  limitations  and  make 
accommodations  beforehand  to  account  for  them.  These  limitations  should  be  included 
in  the  test  instructions  by  the  test  administrators. 

The  static  nature  of  the  final  PAT  as  a  written  test  makes  it  susceptible  to 
compromise  via  memorization  or  copying  and  dissemination.  While  outside  the  scope  of 
the  research  project,  the  final  version  of  the  test  should  be  a  computerized  version  of  the 
PAT  that  can  be  easily  transported,  administered,  taken,  graded,  and  evaluated.  The 
precursor  to  such  a  digitized  version  is  the  written  PAT  presented  in  this  research  project. 
This  product  has  written  scenarios  and  accompanying  questions  that  satisfy  the  task, 
conditions,  and  performance  steps  in  the  SMCT.  The  digitized  version  will  have  to 
present  these  questions  to  each  scenario  in  a  random  manner.  This  random  presentation 
will  limit  test  memorization  and  compromise  and,  by  extension,  its  exploitation  by 
would-be  test  takers. 

The  PAT  developed  in  this  research  is  designed  to  test  proficiency  in  the  36  tasks 
listed  in  the  SMCT.  The  SMCT  signifies  the  fixed  body  of  knowledge  that  the  ECC  has 
determined  that  a  CCO  must  master  to  become  successful  as  a  CCO.  This  PAT  is 
developed  as  a  measure  of  proficiency  of  that  body  of  knowledge  and  is,  therefore,  not 
easily  adaptable  to  emerging  trends  in  contingency  contracting.  In  researching  the 
previously  trained  on  CCO  tasks,  we  found  a  varying  degree  of  recommended  tasks  for 
CCO  training  over  the  past  15  years,  many  of  which  are  not  consistent  with  the  36  tasks 
in  the  SMCT  (Tigges  &  Snyder,  1993;  Lasch,  2002;  Kirstein,  2003).  As  a  result,  we 
make  the  assumption  that  as  the  nature  of  warfare  changes,  so  too  will  the  nature  of 
contingency  contracting  change.  It  is  possible  that  the  tasks  specified  in  the  SMCT  may 

change,  over  time,  with  what  CCOs  face  as  the  nature  of  warfare,  regulations,  and 
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deployments  change.  The  fixed  tasks  in  the  PAT  may  become  outdated  and  need 
revision.  It  is  not  within  the  scope  of  this  project  to  evaluate  or  validate  the  36  tasks 
listed  in  the  SMCT.  ECC  leaders,  PAT  administrators,  and  CCOs  should  keep  in  mind 
the  dynamic  nature  of  contingency  operations  and  their  contracting  needs.  The  research 
team  recommends  that  the  PAT  become  an  evolving  instrument,  the  content  of  which  is 
managed  in  some  form  by  the  ECC.  After  its  inclusion  as  a  training  tool  for  the  ECC, 
obvious  or  glaring  omissions  not  tested  should  be  identified  by  the  ECC  for 
incorporation,  and  extraneous  tasks  tested  should  be  identified  for  exclusion  in  future 
editions  of  the  PAT.  Periodic  reviews  and  doctrinal  revisions  of  the  SMCT  should 
similarly  trigger  PAT  revisions.  In  the  interim,  ECC  leaders  and  CCOs  should  be  aware 
of  emerging  trends  and  develop  solutions  to  test  for  tasks  not  listed,  or  omit  tasks  from 
the  test  that  are  no  longer  relevant. 

Contingency  contracting  is  a  skill  that  takes  years  to  develop.  An  individual’s 
development  consists  not  just  of  completing  educational  requirements  but  also  of  gaining 
experience  to  make  decisions  in  a  dynamic  and  complex  environment.  Contingency 
contracting  is  as  much  an  art  as  it  is  a  science.  It  is  making  the  most  appropriate 
decisions  among  a  multitude  of  scenarios  and  variables  that  makes  for  a  well-trained 
CCO.  Whether  in  written  format  or  as  a  computerized  version,  no  form  of  test,  unless  it 
is  performance-based,  can  accurately  capture  and  measure  the  “art”  of  contingency 
contracting,  developed  over  years  of  experience.  No  written  or  computerized  test  can 
reproduce  the  interface  and  interaction  required  to  accomplish  many  of  the  36  SMCT 
tasks.  The  PAT  is  limited  in  this  respect.  The  research  team  recognizes  that  it  cannot 
fully  measure  a  CCO’s  level  of  proficiency.  However,  the  PAT  can  be  developed  to 
measure  a  baseline  of  knowledge  in  the  36  tasks  by  presenting  scenarios  and  questions 
from  which  the  CCO  must  choose  the  right  answer.  Additionally,  the  PAT  was 
developed  by  the  research  team  to  measure  a  certain  level  of  proficiency  by  creating 
scenarios  and  accompanying  questions  that  will  trigger  experienced  CCOs  to  search  for 
the  “best  answer.” 
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G. 


METHODOLOGY 


The  research  team  began  the  research  by  conducting  a  literature  review  of  all 
pertinent  contingency-contracting  literature.  Contingency  contracting  is  a  niche  subject 
area  in  the  greater  realm  of  contract  management,  executed  almost  exclusively  by  the 
Department  of  Defense  (DoD).  As  such,  we  found  very  few  published  articles  in 
established  scholarly  journals,  practitioner  journals,  general-interest  magazines,  or  other 
publications.  Moreover,  the  specific  nature  of  the  project,  to  develop  a  test  for  CCOs, 
further  narrowed  the  scope  of  applicable  journals.  The  databases  of  the  Defense 
Technical  Information  Center  (DTIC)  and  Inside  Defense  provided  the  majority  of 
articles  germane  to  the  research  topic.  The  research  team  reviewed  all  pertinent  and 
current  articles  covering  contingency  contracting  to  provide  context  in  the  development 
of  the  test. 

The  literature  review  chapter  covers  the  history  of  contingency  contracting  before, 
and  its  evolution  since,  Operation  Desert  Stonn  (ODS).  This  evolution  encompasses  the 
drawdown  of  much  of  the  Army’s  acquisition  workforce  during  the  early  1990s  and  the 
near  simultaneous  increase  in  operational  requirements  during  that  time  following  the 
terrorist  attacks  of  September  11,  2001.  Furthermore,  the  literature  review  covers  the 
resultant  fallout  of  these  changes,  culminating  in  the  Gansler  Commission  findings  and 
recommendations.  By  understanding  contingency  contracting  from  a  historical 
perspective,  the  research  team  established  a  context  for  developing  the  PAT. 

Critical  to  the  literature  review  was  the  full  understanding  of  the  ECC  as  an 
organization.  As  the  research  sponsor  and  customer,  the  ECC  required  a  final,  usable 
PAT  that  would  support  their  needs.  This  includes  understanding  the  ECC’s  role  and 
purpose,  command  structure,  subunits,  training  methodology,  and  where  in  its  growth-to- 
completion  the  ECC  stands.  The  ECC  was  created  in  direct  response  to  the  Gansler 
Commission’s  recommendations  to  prepare,  train,  and  provide  skilled  contracting 
personnel  in  support  of  U.S.  Army  contingency  operations  (Gansler,  2007).  It  is  a  newly 
created  Army  organization  under  command  of  a  one-star  general.  Figure  8  shows  the 
ECC’s  organization  under  the  Army’s  Contracting  Command.  At  the  time  of  this 

writing,  the  ECC,  as  a  new  and  large  organization,  is  in  the  midst  of  growing  and  has  yet 
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to  reach  its  final  end-strength.  It  was  important  for  us  to  understand  where  the  ECC 
stood  in  terms  of  its  development  as  an  organization  in  order  to  understand  how  the  PAT 
complemented  the  ECC’s  training  focus  and  methodology  and  how  the  PAT  could 
remain  relevant  and  useful  over  time. 

The  SMCT  represents  the  cornerstone  of  this  research  project  since  this  single 
document  represents  the  body  of  knowledge  upon  which  the  PAT  was  developed.  The 
SMCT  lists  the  36  tasks  and  performance  steps  to  successfully  accomplish  each  task.  The 
performance  steps  are,  in  essence,  the  correct  answer  for  each  accompanying  task.  The 
SMCT  also  articulates  what  role,  in  the  greater  scheme  of  Army  training  doctrine,  the 
SMCT  fills.  In  developing  the  PAT,  it  was  critical  for  us  to  understand  the  tasks  within 
the  SMCT,  how  their  proficiency  is  measured  in  the  SMCT,  and  how  this  overall 
document  fits  in  the  ECC’s  training  methodology. 

Additionally,  no  official  test  to  augment  a  U.S.  Army  unit’s  training  methodology 
would  be  complete  without  a  thorough  review  of  Army  training  doctrine.  The  research 
team  reviewed  US  Army  Field  Manuals  (FM)  7-0(Training  the  Force)  (2002),  7-l(Battle 
Focused  Training)  (2003),  and  Army  Regulation  (AR)  350-l(Army  Training  and  Leader 
Development) .  Collectively,  these  three  documents  represent  the  cornerstone  of  U.S. 
Army  training  doctrine  and  methodology,  and  we  used  them  to  develop  the  PAT  in 
context  as  a  formal  supplement  to  the  SMCT. 

Lastly,  the  research  team  conducted  a  review  of  testing  and  assessment  theories. 
This  review  included  theoretical  application  of  assessment  as  a  subset  of  the  adult¬ 
learning  model,  core  test  principles,  test  construction  and  test  blueprints,  Bloom’s 
Taxonomy,  and  item  writing  as  it  relates  to  test  construction.  The  knowledge  we  gained 
in  reviewing  test  construction  and  evaluation-practices  literature  allowed  us  to  choose  the 
best  practices  to  develop  a  valid  and  reliable  proficiency-assessment  test  in  a  time- 
constrained  environment  that  meets  the  needs  of  the  Expeditionary  Contracting 
Command  and  U.S.  Army  CCOs. 
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In  developing  the  PAT,  the  research  team  took  each  task  and  broke  it  down  to  its 
subcomponents  of  task,  conditions,  and  performance  steps.  The  tasks  in  the  SMCT 
represent  the  36  testable  tasks  of  PAT  that  we  used  to  measure  proficiency.  The 
conditions  for  each  task  “identify  all  the  equipment,  tools,  references,  job  aids,  and 
supporting  personnel  that  the  Soldier  needs  to  use  to  perfonn  the  task  in  wartime”  (DoA, 
2009,  September  22,  p.  1-3).  The  performance  steps  in  the  SMCT  are  essentially  the 
correct  way  to  demonstrate  proficiency.  They  are,  in  effect,  the  answers  to  the  test.  By 
using  the  task,  conditions,  and  performance  steps  in  the  SMCT,  the  research  team  reverse- 
engineered  each  of  the  36  tasks  in  the  SMCT  to  create  scenarios  and  questions  for  the 
answers  of  the  test. 

Once  we  developed  a  scenario  or  group  of  scenarios  and  their  accompanying 
questions  and  answers,  we  sent  the  scenarios  electronically  to  the  ECC  for  vetting.  The 
research  team’s  sponsor  and  point  of  contact  at  the  ECC  distributed  each  of  the  scenarios 
to  varying  groups  of  experienced  and  senior  leaders  within  the  ECC.  The  research  team’s 
sponsor  and  the  group  of  vetting  officers  are  all  experienced  CCOs.  This  group 
developed  the  SMCT  for  the  MOS  5 1C.  Additionally,  they  vetted  the  scenarios  to  ensure 
that  they  satisfied  the  requirements  for  each  of  the  tasks  listed  in  the  SMCT.  If  any  of  the 
tasks  needed  revision,  the  vetting  group  submitted  any  recommended  changes  to  our 
sponsor,  who,  in  turn,  submitted  them  to  the  research  team.  The  research  team  had  no 
direct  contact  or  correspondence  with  the  vetting  group  of  officers.  All  scenario 
submissions,  changes,  and  revisions  were  handled  through  our  sponsor  on  behalf  of  the 
research  team.  Upon  completion  of  all  36  tasks,  the  research  team  compiled  the  36 
scenarios  with  accompanying  questions  and  answers  on  a  single  Word  document.  While 
not  a  part  of  the  scope  of  this  project,  the  research  team  will  continue  to  work  towards 
converting  the  written  version  of  the  PAT  to  a  computerized  assessment  test. 

H.  ORGANIZATION  OF  REPORT 

This  report  is  organized  in  the  following  manner.  Chapter  I,  Introduction, 
provides  information  on  the  background  of  this  research  project,  lists  the  problem 
statement  and  research  objectives,  defines  the  scope  and  deliverables  of  the  project,  lists 


10 


potential  benefits  and  limitations  of  the  project,  and  describes  the  methodology  by  which 
the  research  was  conducted.  Chapter  II,  Literature  Review,  covers  all  pertinent  articles 
from  scholarly  journals,  practitioner  journals,  general-interest  magazines,  and  other 
publications  on  the  topics  of  training  and  evaluating  contingency  contracting  and  on  test 
construction  and  knowledge  assessment.  Chapter  III,  Contingency-Contracting 
Framework,  looks  in  detail  at  the  organization  responsible  for  the  SMCT,  the 
Expeditionary  Contracting  Command,  and  covers  its  history,  organizational  structure,  and 
training  focus.  Additionally,  this  chapter  describes  in  detail  the  SMCT,  the  document  that 
represents  the  body  of  knowledge  for  the  PAT.  Chapter  IV,  PAT  Development,  describes 
the  steps  the  research  team  took  in  developing  the  PAT.  Lastly,  Chapter  V,  Conclusion, 
summarizes  the  research  project  and  makes  recommendations  for  areas  of  further 
research. 

I.  SUMMARY 

U.S.  Army  CCOs  have  been  and  continue  to  be  a  critical  force  multiplier  in  any 
contingency  operation.  Changes  in  the  force  structure  of  U.S.  Anny  acquisition,  coupled 
with  increasing  demands  on  CCOs  due  to  the  changing  nature  of  warfare,  led  to  a 
shortage  of  a  qualified  and  capable  CCO  base  for  the  U.S.  Army.  This  shortage  caused 
the  Army  to  rely  heavily  on  sister  Services  to  provide  CCOs  (Gansler,  2007).  In  response 
to  the  Gansler  Commission  findings,  and  in  an  effort  to  better  support  the  warfighter,  the 
U.S.  Army  established  the  Expeditionary  Contracting  Command  in  Fort  Belvoir, 
Virginia.  This  new  command,  under  a  one-star  general  leadership,  is  tasked  with 
“providing  skilled,  trained,  contracting  personnel  for  the  support  of  expeditionary  forces” 
(Gansler,  2007,  p.  52). 

This  new  command,  and  other  resultant  changes  of  the  Gansler  Commission’s 
recommendations,  resulted  in  a  planned  net  increase  of  over  400  uniformed  contracting 
officer  positions  (DoA,  2009a).  Many  of  these  positions  were  for  CCOs,  many  of  whom 
have  no  previous  training  in  their  new  field  of  contracting,  MOS  5 1C.  To  standardize 
training  across  the  ECC  for  all  CCOs,  the  ECC  developed  the  MOS  5 1C  SMCT  as  the 
body  of  knowledge  needed  by  CCOs  to  be  successful  contracting  officers  during 
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deployment.  While  this  was  a  significant  step  in  formalizing  the  ECC’s  training 
methodology,  no  method  of  assessing  performance  of  these  tasks  existed. 

The  research  team,  in  partnering  with  the  ECC,  developed  the  Contingency 
Contracting  Officer  Proficiency-assessment  Test  (PAT).  The  PAT  is  based  on  the  36 
tasks  listed  in  the  SMCT.  The  PAT  consists  of  a  scenario  or  scenarios  for  each  of  the  36 
tasks.  Each  scenario  has  accompanying  questions  that  measure  Bloom’s  cognitive 
domains  of  remembering,  understanding,  and,  to  a  limited  extent,  application  of 
knowledge  of  the  tasks  as  defined  by  the  performance  steps.  Proficiency  of  each  task  is 
demonstrated  by  successfully  answering  questions  that  meet  the  criteria  in  the 
performance  steps. 

The  written  version  of  the  PAT  has  several  limitations.  As  a  static,  written  test,  it 
is  vulnerable  to  compromise  and  dissemination.  Moreover,  because  it  is  based  on  the  36 
tasks  in  the  SMCT,  it  is  not  easily  adaptable  to  emerging  trends  in  contingency 
contracting.  Lastly,  a  written  test  can  never  fully  capture  the  “art”  of  contingency 
contracting.  The  learning  curve  for  a  CCO  is  too  long  and  the  variables  too  diverse  to 
fully  capture  the  nuance  of  possible  choices  to  a  “right”  answer.  Despite  these 
limitations,  a  PAT  will  serve  a  critical  role  in  the  Army  and  ECC  training  methodology 
by  providing  the  feedback  mechanism  to  CCOs,  ECC  leaders,  commanders,  and  trainers 
responsible  for  the  professional  development  of  CCOs. 

The  PAT  developed  in  this  research  project  will  serve  to  augment  the  ECC’s 
training  methodologies  by  providing  a  critical  need  in  validating  task  proficiency  in  the 
36  tasks  listed  in  the  SMCT.  While  not  within  the  scope  of  this  project,  a  digitized 
version  of  the  PAT  will  make  the  PAT  easily  transferable  and  accessible.  The  products 
of  this  research  project  will  provide  a  critical  feedback  to  ECC  CCOs  and  their  leaders  by 
providing  feedback  on  training  and  experience.  The  PAT  will  not  only  help  the  ECC 
continue  to  grow  as  an  organization  but  also  help  it  grow  its  newly  assessed  CCOs  into 
competent  and  capable  CCOs  who  serve  as  force  multipliers  for  U.S.  Army  warfighters 
in  current  and  future  contingencies. 
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The  next  chapter,  Literature  Review,  will  cover  all  pertinent  articles  from 
scholarly  journals,  practitioner  journals,  general-interest  magazines,  and  other 
publications  on  the  topics  of  training  and  evaluating  contingency  contracting  and  on  test 
construction  and  knowledge  assessment.  With  respect  to  contingency  contracting,  the 
literature  review  focuses  on  the  history  of  contingency  contracting  and  previous  training 
tasks  to  train  CCOs.  In  the  realm  of  test  construction,  the  research  team  focused  on  the 
theoretical  and  academic  study  of  the  adult-leaming  model,  core  test  principles,  the 
practical  application  of  test  blueprints,  item  writing,  and  test  construction,  all  within  the 
construct  of  Bloom’s  Taxonomy. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

Developing  a  proficiency-assessment  test  for  contingency  contracting  officers 
brings  together  two  highly  specialized  and  disparate  fields:  contingency  contracting  and 
test  construction  and  evaluation.  Contingency  contracting  is  a  niche  subject  area  within 
the  greater  realm  of  defense  contracting.  This  literature  review  focuses  on  the  history  of 
contingency  contracting,  with  an  emphasis  primarily  on  training  and  evaluation  of  CCOs. 
The  research  team  wanted  to  understand  trends  and  changes  in  CCO  training  and 
evaluation  throughout  modern  military  history  in  order  to  help  build  an  effective 
proficiency-assessment  test.  We  understood  that  to  develop  a  proficiency-assessment  test 
would  require  extensive  knowledge  of  test  construction  theory  and  practices.  This 
research  focuses  on  the  theoretical  and  academic  study  of  the  adult-learning  model,  core 
test  principles,  practical  application  of  test  blueprints,  item  writing,  and  test  construction 
within  the  construct  of  Bloom’s  Taxonomy.  The  research  team  used  this  knowledge  to 
create  the  most  effective  and  efficient  proficiency-assessment  test  for  the  36  tasks  in  the 
SMCT,  given  the  limited  time  and  expertise  constraints  of  the  research  team. 

A  thorough  review  of  related  literature  revealed  an  abundance  of  research  efforts 
and  published  articles  on  contingency-contracting-related  topics.  However,  this  literature 
review  concentrates  on  the  evolution  of  U.S.  Army  contingency  contracting  as  it  relates 
to  training,  assessing,  and  preparing  CCOs  for  deployment  to  hostile  environments,  a 
narrower  field  of  study  with  less  available  research  literature.  Areas  covered  include  the 
following:  the  history  of  contingency  contracting,  the  problems  and  challenges  of 
contingency  contracting,  the  history  of  established  tasks  related  to  contingency 
contracting,  and  the  evolution  of  contingency  contracting  since  9/11.  By  understanding 
contingency  contracting  from  a  historical  perspective,  the  research  team  established  a 
context  for  developing  the  PAT.  Moreover,  by  reviewing  the  historical  training  tasks  for 
CCOs,  the  research  team  gained  an  understanding  of  the  SMCT  as  the  base  document  for 
the  development  of  the  PAT. 
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B.  CONTINGENCY  CONTRACTING 

U.S.  Army  contingency  contracting  is  a  highly  functional  and  narrowly  focused 
area  of  study.  As  a  result,  there  is  little  relevant  literature  to  be  extracted  from  civilian 
scholarly  journals,  practitioner  journals,  general-interest  magazines,  or  other  publications. 
The  research  team  reviewed  the  National  Contract  Management  Association’s  database 
for  relevant  articles.  Comparable  studies  that  specifically  addressed  the  issue  of 
contingency-contracting  training  were  very  limited.  The  research  team  reviewed 
databases  of  the  Office  of  the  Inspector  General  with  similar  results.  The  databases  of  the 
Defense  Technical  Information  Center  (DTIC)  and  Inside  Defense  proved  to  be  the  best 
research  databases  to  search  for  contingency-contracting-related  topics.  A  search  of 
those  two  databases  provided  several  contingency-contracting-related  studies  conducted 
by  graduate  students  at  the  Naval  Postgraduate  School  (NPS)  and  the  Air  Force  Institute 
of  Technology  (AFIT).  These  studies — along  with  Government  Accountability  Office 
(GAO)  reports,  the  Gansler  Commission  report,  and  DoD  and  Air  Force  guidance — are 
the  foundation  of  this  portion  of  the  literature  review  and  study. 

1.  History  of  Contingency  Contracting  Pre-9/11 

Contingency-contracting  officers  have  played  an  important  role  throughout 
modem  military.  The  origins  of  contingency  contracting  in  the  U.S.  military  can  be 
traced  back  to  the  American  Civil  War  (Kirstein,  2003).  Mason  (1988)  provided  a 
timeline  outlining  the  role  of  procurement  personnel  from  that  time  period  to  2003. 
Contracting  officers  of  the  1860s,  then  called  quartermasters,  were  directly  assigned  to 
anny  divisions  fighting  the  Civil  War.  As  an  integral  part  of  the  force,  the  Quartermaster 
corps  reduced  the  need  for  a  logistical  footprint  to  support  the  war  effort  (Mason,  1988). 

During  World  War  I  and  World  War  II,  contingency  contracting  personnel  played 
a  lesser  role  because  most  supplies  were  shipped  from  the  United  States  (Kirstein,  2003). 
Shipping  supplies  from  the  United  States  by  watercraft  proved  both  slow  and  inflexible  to 
the  needs  of  the  warfighter.  Goods  often  arrived  long  after  the  need  for  them  had  passed. 
This  resulted  in  surpluses  in  some  items  and  severe  shortages  in  others.  On  a  limited 
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scale,  the  British  carried  out  contracting  tasks  by  providing  surplus  goods  to  supplement 
those  arriving  from  the  United  States  (Mason,  1988). 

During  the  Korean  War,  contracting  was  used  almost  exclusively  for  providing  all 
classes  of  supplies  for  the  troops,  marking  the  first  time  the  troops  relied  on  contracting 
as  an  important  procurement  vehicle.  The  ability  of  local  vendors  to  provide  basic  items 
reduced  the  need  for  shipments  from  the  United  States.  Japanese  and  Korean  contractors 
provided  extensive  support  to  the  war  effort.  Local  procurement  of  supplies  resulted  in  a 
flexible  supply  chain  that  was  responsive  to  the  warfighter’s  needs  (Kirstein,  2003). 
During  the  Vietnam  conflict,  the  utilization  of  contracts  to  procure  supplies  and  services 
greatly  declined.  Mason  attributes  this  decline  to  the  fact  that  a  formal  war  declaration 
was  never  enacted  by  the  United  States  Congress.  Without  a  formal  declaration  of  war, 
the  Air  Force  did  not  have  the  political  support  or  resources  it  needed  to  provide  optimal 
support  (Mason,  1988). 

During  Operation  Desert  Storm  and  Desert  Shield  (ODS),  contingency 
contracting  as  we  know  it  today  was  still  maturing.  Contingency-contracting  personnel 
were  able  to  provide  unprecedented  support  to  the  warfighters  (Kirstein,  2003),  but  faced 
many  challenges  as  they  attempted  to  procure  supplies  and  services.  Their  biggest 
challenge  came  from  the  Government’s  own  regulations  (Luse,  Madeline,  Smith  &  Starr, 
2005).  Restrictions  imposed  by  the  Federal  Acquisition  Regulation  (FAR),  Defense 
Federal  Acquisition  Regulation  Supplement  (DFARS),  and  Army  Federal  Acquisition 
Regulation  Supplement  (AFARS)  made  contingency-contracting  operations  more  difficult 
than  they  needed  to  be.  These  regulations  were  not  designed  for  contingency  operations, 
and  their  applicability  under  wartime  conditions  was  vague  (Luse,  2005).  In  his  book 
Moving  Mountains:  Lessons  in  Leadership  and  Logistics  in  the  Gulf  War,  General 
Pagonis  (1992)  described  the  daunting  mission  of  contracting  personnel,  stating  “our 
limited  and  precious  transport  space  should  be  reserved  for  combat  troops,  and  for  these 
supplies,  such  as  weapons  and  ammunition  [...].  Everything  else  was  our  problem  to  be 
found  and  contracted  for”  (Pagonis,  1992,  p.  107).  During  ODS,  most  logistical  needs 
were  met,  at  least  in  part,  by  CCOs.  Items  and  services  contracted  for  included  food, 
water,  lodging,  laundry,  sanitation,  communications,  transportation,  and  miscellaneous 
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equipment  (Pagonis,  1992).  Both  the  Army  and  the  Air  Force  relied  heavily  on  contract 
leases  and  procured  goods  and  services.  The  lessons  learned  from  Operation  Desert 
Storm  shaped  current  CCO  practices  and  are  being  applied  to  meet  the  challenges  of 
Operation  Iraqi  Freedom  and  Operation  Enduring  Freedom  (Kirstein,  2003). 

2.  History  of  Contingency  Contracting  Post-9/11 

The  21st  century  has  brought  about  significant  changes  in  the  manner  in  which 
U.S.  Forces  conduct  operations  overseas.  The  post-Cold  War  reduction  in  forces  saw  a 
decrease  in  acquisition  workforce  professionals  throughout  the  Anny.  From  1990  to 
1996,  the  U.S.  Army  saw  its  acquisition  workforce  decrease  from  approximately  10,000 
personnel  to  approximately  5,500  personnel,  where  it  remained  through  2001  (Gansler, 
2007).  At  the  same  time,  the  United  States  military,  and  in  particular  the  Army, 
significantly  increased  the  outsourcing  of  many  jobs  that  were  previously  performed  by 
uniformed  Service  members.  As  an  indicator  of  complexity  in  contracting,  the  value  of 
U.S.  Army  contracts  increased  331%  from  $23.3  billion  in  1992  to  $100.6  billion  in 
2006,  and  the  number  of  contracts  actions,  as  an  indicator  of  workload,  increased  654% 
from  approximately  52,900  to  approximately  398,000  over  the  same  timeframe  (Gansler, 
2007).  This  decrease  in  contracting  professionals  and  increase  in  contracting 
requirements  led  to  many  deficiencies  in  the  Army’s  ability  to  conduct  procurements  for 
its  warfighters  during  contingency  operations. 

Operation  Iraqi  Freedom  (OIF)  and  Operation  Enduring  Freedom  (OEF)  have 
seen  the  largest  use  of  contingency  contracting  by  the  Department  of  Defense  in  its 
history.  Operations  in  Iraq  and  Afghanistan  have  demonstrated  the  demand  that 
expeditionary  military  operations  place  on  the  contracting  system  and  contracting 
personnel.  A  2008  GAO  report  analyzed  data  from  the  DoD,  Department  of  State,  and 
United  States  Agency  for  International  Development  (USAID)  on  contracting  activities  in 
Iraq  and  Afghanistan  for  fiscal  year  (FY)  2007  and  the  first  half  of  FY  2008.  This  report 
reviewed  (1)  the  number  and  value  of  contracts  and  the  extent  they  were  awarded 
competitively,  (2)  the  number  of  contractor  personnel,  including  those  perfonning 
security  functions,  and  (3)  the  number  of  contractor  personnel  who  were  killed  or 
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wounded.  The  report  showed  that  the  United  States  had  more  civilian  contractor 
personnel  in  Iraq  in  2008  than  it  did  U.S.  Army  soldiers.  These  contractor  employees 
provide  food  services,  interpretation  services,  communications,  equipment  repair,  and 
other  important  services. 

3.  Problems  and  Challenges  in  Contingency  Contracting 

The  shortfall  in  trained  U.S.  Army  contracting  personnel  in  Iraq  and  Afghanistan 
and  the  exponential  increase  in  contingency-contracting  procurement  requirements  led  to 
failures  highlighted  in  the  2007  report  by  the  Gansler  Commission.  Officially  called 
Urgent  Reform  Required:  Army  Expeditionary  Contracting,  the  Commission  on  Anny 
Acquisition  and  Program  Management  in  Expeditionary  Operations  (led  by  Dr.  Jacques 
S.  Gansler,  former  Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics)) 
examined  the  Army’s  business  processes  as  they  relate  to  expeditionary  contracting 
operations  and  highlighted  the  shortfalls  and  problems  experienced  by  the  Army  prior  to 
October  2007.  The  report  found  that  some  critical  segments  of  the  “Institutional  Anny” 
had  not  adapted  to  the  expeditionary  acquisition  requirements,  listing  these  “key  failures” 
as  “significantly  contributing  to  the  fraud,  waste,  and  abuse  in-theater”  by  Army 
personnel  (Gansler,  2007,  p.  1).  Of  these  critical  segments,  two,  training  and  education 
and  doctrine,  regulations  and  processes,  are  addressed  in  some  measure  by  our 
development  of  the  proficiency-assessment  test.  The  PAT  is  nested  in  newly  developed 
doctrine,  as  found  in  the  SMCT.  Moreover,  it  provides  a  critical  feedback  mechanism  for 
CCOs  and  their  commanders  on  CCO  training  plans. 

The  Gansler  Commission  recommended  four  overarching  changes  for  the  Army: 

1 .  Increase  the  stature,  quantity,  and  career  development  of 
military  and  civilian  contracting  personnel  (especially  for 
expeditionary  operations). 

2.  Restructure  organization  [Army]  and  restore  responsibility  to 
facilitate  contracting  and  contract  management  in  expeditionary 
and  CONUS  [Continental  United  States-based]  operations. 

3.  Provide  training  and  tools  for  overall  contracting  activities  in 
expeditionary  operations. 
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4.  Obtain  legislative,  regulatory,  and  policy  assistance  to  enable 
contracting  effectiveness  in  expeditionary  operations.  (Gansler, 

2007,  p.  5) 

The  establishment  of  the  Expeditionary  Contracting  Command  is  a  direct  result  of 
recommendation  number  two  above.  The  ECC’s  development  of  the  SMCT  as  a  standard 
of  knowledge  for  CCOs  is  in  direct  response  to  the  Gansler  Commission’s  third 
recommendation.  In  the  Gansler  report,  the  Commission  further  elaborates  on  their  third 
recommendation  by  adding  “Develop  and  field  contract  ‘tools’  needed  for  expeditionary 
forces  including,  but  not  limited  to,  sample  contracts,  statements  of  work,  pre-agreed-to 
waivers,  etc.”  (Gansler,  2007,  p.  55).  It  is  this  recommendation  that  necessitates  the 
development  of  the  SMCT  and,  by  extension,  a  proficiency-assessment  test  to  measure 
against  the  standard  of  the  SMCT. 

While  the  Gansler  Commission  is  the  most  widely  referenced  literature  pertaining 
to  failures  in  the  Army’s  acquisition  workforce,  similar  calls  for  better  training  of 
personnel  had  been  highlighted  by  other  governmental  oversight  agencies  as  well.  In 
July  2007,  David  M.  Walker,  then  Comptroller  General  of  the  United  States,  gave 
testimony  before  the  Committee  on  Homeland  Security  and  Governmental  Affairs  of  the 
U.S.  Senate,  highlighting  systemic  challenges  to  federal  acquisition  and  contracting.  His 
statement,  published  in  GAO  Report  07-1098T,  Federal  Acquisition  and  Contracting: 
Systemic  Challenges  Need  Attention,  highlights  four  key  acquisition  challenges  affecting 
agencies  in  the  U.S.  Government.  These  challenges  are  as  follows:  (1)  separating  wants 
from  needs,  (2)  establishing  and  supporting  realistic  program  requirements,  (3)  using 
contractors  in  appropriate  circumstances  and  contracts  as  a  management  tool,  and  (4) 
creating  a  capable  workforce  and  holding  it  accountable  (GAO,  2007).  A  proficiency- 
assessment  test  will  assist  the  ECC  in  creating  a  capable  workforce  by  testing  their 
proficiency  against  an  established  standard. 

While  the  Anny  has  gone  to  great  lengths  to  correct  the  deficiencies  highlighted 

in  the  Gansler  report  (DoA,  2009a),  many  problems  in  the  training  of  contracting 

personnel  exist.  For  example,  in  March  2008,  the  GAO  investigated  the  Army 

Contracting  Agency’s  (ACA)  Contracting  Center  of  Excellence  (CCE)  to  better 

understand  the  use  of  contractors  as  contract  specialists.  The  primary  concern  of  the 
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GAO  was  that  the  Army’s  practice  of  hiring-out  contract  specialists  placed  risk  on  the 
Army  of  entering  into  personal  service  contracts.  The  GAO  identified  that  the  CCE  did 
not  have  a  formal  training  program  for  its  pennanent  government  contracting  specialists. 
CCE  officials  acknowledged  the  lack  of  a  training  program  and  the  need  to  provide  better 
training  to  their  contracting  specialists  (GAO,  2008,  March). 

A  similar  study  highlighting  the  lack  of  training  in  contracting  personnel  was 
published  in  June  2009  by  the  Commission  on  Wartime  Contracting  (CWC).  The  CWC 
was  recently  established  by  Congress  to  investigate  fraud,  waste,  and  abuse  allegations 
surrounding  contingency  contracting  in  a  wartime  environment.  The  CWC  provided  an 
interim  report,  titled  At  What  Cost?  Contingency  Contracting  in  Iraq  and  Afghanistan, 
that  acknowledged  the  long-standing  training  problems  of  the  federal  civilian  and 
military  contracting  workforce.  The  report  states: 

The  ongoing  analysis  includes  a  review  of  the  reports  related  to 
contingency  contracting  published  between  FY  2003  and  FY  2009.  A 
number  of  themes  surface  in  these  reports,  such  as  staffing  issues, 
inadequate  internal  controls,  and  the  need  for  more  training  of  personnel 
with  contracting  responsibilities.  (CWC,  2009,  p.  26) 

The  repeated  call  for  more  and  better  training  for  CCOs  by  government  oversight 
agencies  appears  to  be  more  prevalent  due  to  the  wars  in  Iraq  and  Afghanistan.  However, 
a  literature  review  of  previous  training  plans  and  CCO  training  tasks  reveals  that  the 
development  of  CCO  training  has  evolved  with  our  military’s  missions  over  the  past 
decades.  The  next  portion  of  the  literature  review  focuses  on  recommended  CCO 
training  tasks,  which  culminated  with  the  development  of  the  U.S.  Anny’s  Soldier’s 
Manual  of  Common  Tasks. 

4.  Formalization  of  Contingency  Contracting  Officer  Training  and 
Tasks 

Until  recently,  the  U.S.  military  lacked  a  formalized  training  program  that 
adequately  prepared  CCOs  for  contingency  contracting  operations.  In  recent  years, 
studies  have  identified  the  need  for  fonnalized  training  and  have  presented 
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recommendations  on  the  tasks  in  which  CCOs  need  training  to  perform  that  will  best 
prepare  them  for  deployment  to  a  contingency  environment. 

Mason  (1988)  identified  the  need  for  formal  CCO  training  in  his  research.  Using 
qualitative  data  derived  from  interviewing  experienced  contracting  personnel,  Mason 
investigated  what,  where,  and  how  contingency  contracting  training  was  conducted  in  the 
past.  Mason  investigated  existing  legislation,  regulations,  and  policies  that  affected  the 
ability  of  CCOs  to  perform  their  duties,  and  he  looked  at  potential  changes  that  could  be 
made  to  existing  training  methods  to  aid  CCOs  in  carrying  out  contingency  support 
missions.  Even  though  Mason  determined  that  there  was  a  legitimate  need  for  CCOs  to 
be  trained  by  their  units  prior  to  deployment,  he  did  not  present  a  formal  training 
program.  Instead,  he  presented  a  list  of  recommendations  that  focused  on  institutional, 
procedural  and  policy  changes  to  better  support  contingency  contracting. 

Killen  and  Wilson  (1992)  recommended  the  following  topics  for  CCOs  deploying 
with  their  units: 

1 .  Overseas  acquisition  procedures 

2.  Local  purchasing 

3.  Alternative  methods  of  contracting 

4.  The  economic  variation  provision  for  anned  conflict 

5.  Base  closure  at  the  end  of  conflict 

This  list  was  the  first  of  its  kind  in  that  it  recommends  special  areas  of  emphasis  in  order 
for  CCOs  to  be  successful  in  a  contingency  environment.  The  list,  as  limited  as  it  may 
appear  today,  provided  a  foundation  for  building  realistic  training  programs  for  future 
CCOs  (Kirstein,  2003). 

Tigges  and  Snyder  (1993)  conducted  further  research  in  the  field  of  contingency 
contracting.  They  presented  a  list  of  recommended  CCO  training  topics  based  on  the 
results  of  their  study.  Their  study  focused  on  the  impressions  CCOs  had  of  the  training 
they  received  prior  to  deployment.  They  conducted  a  survey  and  several  interviews  to 
collect  data.  Their  population  of  interest  was  CCOs  who  deployed  during  Operation 
Desert  Shield.  Tigges  and  Snyder  compiled  a  list  of  50  topics  and  ranked  them  in 
descending  order  based  on  their  significance.  The  following  is  a  list  of  CCO  training 
tasks  by  rank,  as  identified  by  Tigges  and  Snyder. 
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1.  Services 

2.  Communications  lines 

3.  Use  of  SF  44,  Purchase  Order  and  Invoice  Voucher 

4.  Currency  issues 

5.  Use  of  Blanket  Purchase  Agreements 

6.  Host  Nation  Support  Agreements 

7.  Commodities 

8.  Obtaining  a  vehicle 

9.  Finance  issues  and  procedures 

10.  Procurement  integrity  in  a  contingency 

1 1 .  Cash  payments 

12.  Dealing  with  kickbacks 

13.  Makeup  of  local  community 

14.  Transportation  issues  and  procedures 

15.  Customs  issues 

16.  How  to  establish  a  pre-deployment  listing  of  critical  requirements 

17.  Security  issues 

18.  Protection  of  funds  under  field  conditions 

19.  Supply  issues  and  procedures 

20.  Claims 

2 1 .  Mutual  support  agreements  with  other  nations 

22.  Civil  engineering  issues  and  procedures 

23.  Understanding  legal  authority  under  emergency  conditions 

24.  Cultural  issues 

25.  Obtaining  interpreters 

26.  Anticipating  customer  requests 

27.  Chain  of  command  in  a  contingency 

28.  Converting  funds 

29.  Methods  of  control 

30.  Terminating  for  convenience 

3 1 .  Finding  a  place  to  work 

32.  Services  issues  and  procedures 

33.  Existing  vendor  source  listings 

34.  Working  with  the  other  branches  of  the  U.S.  anned  forces 

35.  Payments 

36.  Dress  issues 

37.  Referrals  to  other  in- theater  organizations 

38.  Obligation  authority  (AF  Form  616) 

39.  Writing  Statements  of  Work 

40.  Home  base  support 

41.  Working  with  the  U.S.  Embassy 

42.  Restrictive  commanders 

43.  Establishing  review  and  approval  procedures 

44.  Record  disposition 

45.  Legal  limitations  of  decentralization 
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46.  Proximity  to  critical  organizations 

47.  Appropriation  sources 

48.  Contract  surveillance 

49.  Termination  for  default 

50.  Intermediate  base  support 

Some  of  these  topics  are  no  longer  an  issue  to  CCOs,  but  many  have  been  incorporated 
into  training  plans  and  contingency-contracting  courses  offered  to  CCOs  across  all 
branches  of  the  military. 

Nash  (1997)  identified  five  skills  essential  to  a  contracting  officer.  These  five 
skills,  listed  below,  also  apply  in  a  contingency  environment. 

1 . )  Knowledge  of  the  rules  of  the  game 

2. )  Ability  to  exercise  sound  business  judgment 

3. )  Knowledge  of  strategy  and  tactics 

4. )  Knowledge  of  the  market 

5. )  Ability  to  function  successfully 

For  each  skill,  Nash  analyzed  the  training  provided  at  that  time  and  suggested  changes. 
Of  the  five  skills,  Nash  argued  that  only  one,  knowledge  of  the  rules  of  the  game, 
received  adequate  training.  His  study  suggested  that  contracting  training  at  that  time  did 
not  fully  prepare  contracting  officers  or  CCOs  for  their  positions.  Furthermore,  if 
contracting  officers  had  all  five  skills,  then  they  probably  learned  them  on  the  job  and  not 
through  a  training  program  (Nash,  1997). 

Lasch  (2002)  used  a  different  approach  to  conduct  his  study.  He  identified  88 
tasks  that  were  regularly  perfonned  by  CCOs  in  deployed  environments.  He  then 
conducted  a  survey  to  obtain  feedback  from  experienced  CCOs  and  Functional  Area 
Managers  (FAMs)  at  the  major  command  level  (MAJCOM)  from  the  Air  Force.  The  two 
groups  were  asked  to  rank  the  88  tasks  based  on  how  important  they  believed  the  tasks 
were  and  the  frequency  with  which  the  tasks  were  performed.  The  result  was  a 
composite  rank-ordered  list  of  tasks  based  on  inputs  from  the  two  surveyed  populations 
that  considered  both  the  importance  and  frequencies  of  the  tasks  (Lasch,  2002).  Lasch 
recommended  the  top  third  of  the  88  tasks  be  included  in  CCO  training  programs.  The 
following  is  a  list  of  the  top  35  training  tasks  of  the  88  identified  in  the  study  (Lasch, 
2002). 
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1 .  Simplified  Acquisition  Procedures 

2.  AF  Form  9,  Purchase  Request 

3.  Installation  Access  for  Contractors 

4.  Customer  Education  on  Contracting  Policies 

5.  Use  of  Automated  Database/Spreadsheet  to  Record  Purchases 

6.  Contract  Modifications 

7.  SF  44,  Purchase  Order  Invoice  and  Voucher 

8.  Blanket  Purchase  Agreements 

9.  Country  Customs  Procedures 

10.  Contract  Modifications 

1 1 .  Expedited  Contracting  Actions 

12.  SF  1449,  Solicitation/Contract  Order  for  Commercial  Items 

13.  Shipment  of  Supplies  Overseas 

14.  Use  of  Government  Purchase  Card 

15.  Solicit,  Award,  and  Administer  Service  Contracts 

16.  After- Action  Report 

17.  Standing  Up  a  Contracting  Office 

18.  SF  30,  Amendment  of  Solicitation 

19.  Deployment/Contingency  Kit  Contents 

20.  Reviewing  Statement  of  Work/Performance  Work  Statement 

2 1 .  Solicit,  Award,  and  Administer  Commodity  Contracts 

22.  Solicit,  Award,  and  Administer  Construction  Contracts 

23.  Host  Nation  Support  Agreements 

24.  Contract  Closeout 

25.  Payments 

26.  Commander’s  In-brief 

27.  Funding  the  Government  Purchase  Card 

28.  Establishing  Vendor  Base 

29.  AF  Form  616,  Fund  Cite  Authorization 

30.  Status  of  Forces  Agreement 

31.  Working  with  Finance/Defense  Finance  and  Accounting  Services 

32.  Writing  Statement  of  Work/Performance  Work  Statement 

33.  Ethics  Training 

34.  Terminations 

35.  Gratuity  Training 

There  is  no  evidence  that  Lasch’s  list  was  incorporated  into  any  CCO  training  program. 
The  list  addresses  the  concerns  and  challenges  CCOs  had  at  that  time  and  provides 
relevant  training  topics,  some  of  which  are  valid  for  the  current  contingency  environment. 

Kirstein’s  (2003)  work  is  follow-on  research  to  Lasch’s  thesis.  Kirstein’s  study 
investigates  how  units  conduct  training  and  to  what  extent  the  recommended  tasks  are 
being  addressed  in  training.  Kirstein  conducted  interviews  and  surveys  and  reviewed 
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archival  training  logs  to  capture  the  data  for  his  study.  His  study  consisted  of  a  pre-test, 
which  was  conducted  through  interviews,  and  a  two-phase  survey.  Phase  one  described 
the  purpose  of  the  study,  and  phase  two  involved  the  collection  of  surveys  from  the 
population  of  interest. 

Kirstein  obtained  training  logs  and  plans  from  the  participants’  units.  These  logs 
provided  insight  into  the  content  of  training  and  allowed  the  researcher  to  investigate 
which  of  the  recommended  tasks  from  Lasch  (2002)  were  included  in  unit  training 
programs  across  the  Air  Force  (Kirstein,  2003).  Kirstein’s  analysis  found  that  a  number 
of  units  in  some  of  the  commands  did  not  conduct  any  formal  CCO  training,  clearly 
indicating  that  many  CCOs  were  not  receiving  training  in  the  top  30  tasks  identified  by 
Lasch  (2002).  Additionally,  the  units  that  conducted  training  did  so  on  approximately 
seven  of  Lasch’s  30  tasks. 

Kirstein’s  review  of  training  logs  also  revealed  that  unit-level  training  alone  was 
not  preparing  CCOs  for  their  deployments.  These  results  indicated  that  the  content  and 
quality  of  training  was  inconsistent  across  contingency  contracting  units.  The  study 
revealed  that  most  units  had  nearly  complete  control  over  their  own  training  programs. 
While  this  allowed  a  great  deal  of  flexibility,  there  was  no  way  of  ensuring  the  adequacy 
and  consistency  of  training  (Kirstein,  2003).  Responses  to  Kirstein’s  survey  indicated 
that  the  typical  unit  conducted  unit-level  CCO  training  for  one  hour  each  month.  The 
training  consisted  of  lectures  and  was  not  realistic.  Kirstein  recommended  that  the  Air 
Force  both  improve  training  programs  and  provide  relevant  training  material  for  future 
unit-level  CCO  training  programs.  The  following  list  of  tasks  is  a  list  of  Kirstein’s 
recommendations.  Tasks  common  to  both  Lasch’s  and  Kirstein’s  lists  are  depicted  in 
bold  below. 

1.  SF  44 

2.  Blanket  Purchase  Agreements 

3.  Use  of  Government  Purchase  Card 

4.  Cultural  Training 

5.  Contract  Types 

6.  Payments/Funding 

7.  Customer  Education  on  Contract  Policies 

8.  Forms 

9.  CCO  Responsibilities 
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10.  Solicit,  Award,  and  Administer  Construction  Contracts 

11.  AF  Form  9/Purchase  Orders/Requests 

12.  Prioritization 

13.  General  Procedures 

14.  Appendix  CC  Overview 

15.  Working  with  Finance 

16.  Documentation 

17.  Inter-service  Procedures 

18.  Deployment/Contingency  Kit  Contents/Usage 

19.  Solicit,  Award,  and  Administer  Service  Contracts 

20.  Commander’s  In-brief 


The  above  list  depicts  20  of  the  34  tasks  recommended  in  Kirstein’s  study.  His 
recommendations  were  based  on  feedback  from  CCOs  who  had  made  numerous 
deployments  to  contingency  environments;  they  continue  to  be  relevant  today.  When 
combined  with  the  list  of  tasks  recommended  by  Lasch  (2002),  the  list  of  tasks 
recommended  by  Kirstein  provides  the  basics  for  a  good  contingency-contracting  training 
plan.  The  methodologies  were  somewhat  different,  but  the  two  studies  arrived  at  a 
similar  conclusion,  with  almost  half  of  the  recommended  tasks  being  present  in  both 
studies. 


5.  Other  Established  Training  Methods 

The  literature  reviewed  thus  far  pertains  to  studies  conducted  by  members  of  the 
Air  Force  and  the  Army  as  well  as  the  Gansler  Commission  report,  which  addressed  the 
problems  plaguing  contingency  contracting  in  the  Army.  Burger,  Kehr  and  Wobensmith 
(2003)  conducted  a  study  sponsored  by  the  Marine  Corps  Institute  (MCI)  on  how  to  assist 
the  Marine  Corps  in  training  its  contracting  personnel  in  preparation  for  deployment  to  a 
contingency  operation.  Burger,  Kehr  and  Wobensmith,  (2003)  identified  five  causes 
affecting  contingency-contracting  training  in  the  Marine  Corps.  These  causes  are  listed 
below. 


Cause  1.  No  Fonnal  School  Training 

Cause  2.  No  Web-based  or  Self-guided  Study 

Cause  3.  No  Direct  CON  234  Supervision 

Cause  4.  No  In-house  Contingency  Contracting  Training 

Cause  5.  Small  Contracting  Specialist  Population 
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Burger,  Kehr  and  Wobensmith  (2003)  included  elaborate  recommendations  for 
each  one  of  these  causes.  For  Cause  1,  they  recommended  Marine  Corps  CCOs  attend 
the  entry-level  Air  Force  Contracting  School.  At  the  time  of  the  study,  the  Air  Force  was 
the  only  Service  that  had  a  fonnal  technical  school  where  enlisted  personnel  could  be 
trained  in  the  contracting  career  field.  Attendance  to  the  Air  Force  Contracting  School 
would  allow  enlisted  CCOs  to  get  training  in  contract  policies,  best  practices,  and 
guidelines  to  perform  their  jobs.  For  Cause  2,  the  study  group  recommended  the 
development  of  a  contingency-contracting  correspondence  manual  that  they,  in 
conjunction  with  the  MCI,  developed  and  published  as  part  of  their  report.  For  Cause  3, 
the  group  recommended  a  more  proactive  role  to  ensure  that  all  CCOs  attended  CON 
234,  Contingency  Contracting  Course.  They  recommended  that  instead  of  only  tracking 
who  completed  the  course,  the  Logistics  Branch  at  Headquarters  Marine  Corps 
(HQMC) — the  branch  of  the  Marine  Corps  that  oversees  the  USMC  CCOs — should  take 
a  more  centralized  approach  to  managing  the  available  slots  and  ensuring  more  CCOs 
attend  CON  234.  For  Cause  4,  the  group  recommended  the  establishment  of  a  Mobile 
Training  Team  (MTT)  made  up  of  experienced  CCOs.  The  team  would  travel  to  the 
locations  where  the  majority  of  CCOs  were  located  and  provide  contingency  contracting 
training. 


6.  Defense  Acquisition  University  CON  234,  Contingency  Contracting 
Course 

The  Defense  Acquisition  University  (DAU)  provides  acquisition  training  and 
career  management  to  the  contingency  contracting  and  procurement  community.  The 
DAU  mission  is  to  “[pjrovide  practitioner  training,  career  management,  and  services  to 
enable  the  Acquisition  Technology  and  Logistics  (AT&L)  community  to  make  smart 
business  decisions  and  deliver  timely  and  affordable  capabilities  to  the  war-fighter” 
(DAU,  2008).  The  DAU  offers  102  formal  courses,  22  of  which  are  in  the  field  of 
contracting;  however,  only  CON  234  addresses  contingency  contracting. 

CON  234  is  a  9-day  course  intended  to  train  and  develop  skills  in  contingency 
contracting  to  support  joint  forces  in  all  military  operations  (DAU,  2008).  While  some 
CCOs  who  have  attended  CON  234  categorize  it  as  a  great  course  that  goes  a  long  way  in 
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preparing  CCOs  for  deployment  to  a  contingency  environment,  others  complain  that  the 
course  is  too  short  and  too  broad  (Burger  et  ah,  2003). 


7.  Contingency  Contracting  Officer  Handbook 

Contingency  Contracting:  A  Joint  Handbook  for  the  21st  Cen  tury  was  prepared 
by  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (OUSD  (AT&L)),  Defense  Acquisition,  Technology,  and  Logistics 
contingency-contracting  staff;  the  Defense  Acquisition  University;  representatives  from 
the  United  States  Army,  Navy,  Marine  Corps,  and  Air  Force;  and  the  Air  Force  Logistics 
Management  Agency  (DPAP,  2008).  The  CCO  handbook  serves  as  a  guide  to  assist 
contingency-contracting  officers  in  preparing  for  wartime  and  peacetime  deployments  or 
exercises  and  performing  the  duties  of  a  deployed  contingency  contracting  officer. 

8.  Soldier’s  Manual  of  Common  Tasks 

The  Soldier’s  Manual  of  Common  Tasks  (SMCT)  was  created  by  the 
Expeditionary  Contracting  Command  (ECC)  for  soldiers  and  officers  holding  Military 
Occupational  Specialty  (MOS)  5 1C  as  a  way  to  focus  on  specific  tasks  that  will  prepare 
CCOs  in  the  ECC  for  their  wartime  mission.  The  SMCT  contains  an  MOS  training  plan 
that  provides  infonnation  needed  to  plan,  conduct,  and  evaluate  individual  training  (DoA, 
2009,  September  22).  The  SMCT  includes  a  list  of  36  tasks  divided  into  four  subject 
areas  that  are  designed  to  prepare  CCOs  to  accomplish  their  mission  in  any  contingency 
environment.  Chapter  III  of  this  report  describes  the  development  of  the  SMCT  and  the 
establishment  of  the  ECC  in  greater  detail. 

The  establishment  and  standardization  of  training  tasks  for  CCOs  in  the  SMCT 

will  help  subordinate  units  and  individual  CCOs  within  the  ECC  focus  their  training 

efforts  prior  to  deployment.  This  body  of  knowledge  is  a  significant  step  in  standardizing 

the  pre-deployment  training  for  CCOs  and  preparing  them  for  their  mission  once 

deployed.  To  supplement  this  training,  an  appropriate  performance  assessment  tool  is 

needed  to  measure  perfonnance  of  these  tasks.  The  ECC  currently  lacks  a  standardized 

method  of  measuring  individual  skill  and  task  proficiency  in  its  CCOs.  A  written 

proficiency-assessment  test  will  allow  leaders  in  the  ECC  to  measure  individual  task 
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proficiency  against  established  standards,  determine  each  CCO’s  preparedness  to  conduct 
duties  during  deployment,  determine  appropriate  CCO  assignment,  and  develop 
individualized  training  programs  for  CCOs. 

Developing  a  proficiency-assessment  test  requires  extensive  knowledge  of  test- 
construction  theory  and  practices.  The  research  team,  comprised  of  professional  soldiers, 
has  no  academic  background  or  experience  in  test  construction.  However,  training 
soldiers  and  assessing  their  performance  is  a  core  competency  of  any  professional  Army 
officer.  In  this  respect,  the  research  team  has  practical  experience.  In  an  effort  to  round 
out  our  practical  experience  with  grounded  theoretical  and  academic  principles  and  to 
assist  in  the  development  of  the  PAT,  the  research  team  reviewed  U.S.  Army  training 
doctrine,  as  it  relates  to  training  and  assessment.  Additionally,  the  research  team 
conducted  a  review  of  assessment  as  a  subset  of  the  adult-learning  model,  core  test 
principles,  test  construction  and  test  blueprints,  Bloom’s  Taxonomy,  and  item  writing,  as 
it  relates  to  test  construction.  This  knowledge  base  will  assist  the  research  team  in 
developing  a  valid  and  reliable  proficiency-assessment  test  to  meet  the  needs  of  the 
Expeditionary  Contracting  Command  and  U.S.  Anny  CCOs. 

C.  TESTING  AND  EVALUATION 

This  portion  of  the  literature  review  will  primarily  review  test  and  assessment 
theory.  At  the  beginning  of  our  research,  the  research  team  set  out  to  detennine  (1)  what 
is  the  role  of  assessment  in  a  professional  workforce,  (2)  what  are  core  test  principles, 
and  (3)  what  are  the  basic  steps  and  procedures  to  test  construction. 

1.  Training  and  Assessment 

Field  Manual  (FM)  7-0  (Training  the  Force),  FM  7-1  (Battle  Focused  Training), 
and  Army  Regulation  350-1  (Army  Training  and  Feader  Development)  are  three  of  the 
main  documents  that  outline  Anny  training  doctrine.  FM  7-0  establishes  Anny  training 
doctrine  while  FM  7-1  is  designed  to  help  unit  commander’s  develop  and  execute 
training  programs  (DoA,  2002).  The  authors  reviewed  those  documents  to  understand 
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how  this  project  fits  within  the  Army  training  methodology  and  comparative  adult¬ 
learning  theory.  The  Army  training  and  leader-development  model  is  depicted  below  in 
Figure  1. 


Figure  1 .  Army  Training  and  Leader  Development  Model 

(DoA,  2002,  p.  1-6) 


FM  7-0  describes  this  model  in  the  following  manner: 


Leader  Development  is  a  lifelong  learning  process.  The  three 
core  domains  that  shape  the  critical  learning  experiences 
throughout  a  soldier’s  and  leader’s  career  are  the  operational, 
institutional,  and  self-development  domains.  Together,  these 
domains  interact  using  feedback  and  assessment  from  various 
sources  and  methods  to  maximize  warfighting  readiness.  Each 
domain  has  specific,  measurable  actions  that  must  occur  to 
develop  our  leaders.  The  operational  domain  includes  home 
station  training,  combat  training  center  rotations,  joint  training 
exercises,  and  operational  deployments  that  satisfy  national 
objectives.  Each  of  these  actions  provides  foundational 
experiences  for  soldiers,  leaders,  and  unit  development.  The 
institutional  domain  focuses  on  educating  and  training  soldiers 
and  leaders  on  the  key  knowledge,  skills,  and  attributes  required 
to  operate  in  any  environment.  It  includes  individual,  unit  and 
joint  schools,  and  advanced  education.  The  self-development 
domain,  both  structured  and  informal,  focuses  on  taking  those 
actions  necessary  to  reduce  or  eliminate  the  gap  between 
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operational  and  institutional  experiences.  Throughout  this 
lifelong  learning  and  experience  process,  there  is  formal  and 
informal  assessment  and  feedback  of  performance  to  prepare 
leaders  for  their  next  level  of  responsibility.  Assessment  is  the 
method  used  to  detennine  the  proficiency  and  potential  of 
leaders  against  a  known  standard.  (DoA,  2002,  p.  1-5) 

The  proficiency-assessment  tool  developed  in  this  project  can  help  assess  and  coordinate 
individual  training  within  the  institutional  and  self-development  domains  of  the  Army 
training  model. 

Assessment  is  core  to  adult-leaming  processes.  The  adult-learning  model 
includes  four  phases  (Knowles,  Holton,  &  Swanson,  1998).  The  first  phase  is  to 
determine  what  learning  objectives  are  desired  and  to  set  learning  achievement  goals. 
The  second  phase  creates  a  strategy  and  allocates  resources  to  accomplish  the  learning 
objectives.  Step  three  involves  implementing  the  learning  strategy,  and  step  four  assesses 
the  attainment  of  the  learning  goal.  Figure  2  depicts  the  theoretical  foundation  of  adult 
learning. 


Figure  2.  Theoretical  Foundation  of  Adult  Learning 
(Knowles,  Holton  &  Swanson,  1998,  p.  125) 
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Army  training  doctrine  considers  assessment  a  commander’s  responsibility  (DoA, 
2002).  It  is  ultimately  the  commander  who  must  determine  if  her  unit  and  soldiers  are 
prepared  to  accomplish  their  operational  mission.  The  commander  utilizes  multiple 
evaluation  tools  to  determine  and  report  mission  readiness.  These  evaluations  can  be 
classified  as  informal,  formal,  internal,  or  external. 

Assessment  is  a  continuous  process  and  is  considered  both  the  end  and  the 
beginning  of  the  training-management  cycle.  The  Army’s  planning  process  for  training 
is  divided  into  four  steps:  training  assessment,  training  strategy,  commander’s  guidance, 
and  training  plans.  Figure  3  depicts  the  Anny’s  planning  process  for  training. 


Figure  3.  Army  Training  Planning  Process  (Training  Assessment) 

(DoA,  2003,  p.  4-4) 

Before  developing  a  training  plan,  a  commander  evaluates  the  individual  battle 
tasks  that  support  overall  unit  mission  essential  tasks.  Mission  Essential  Task  Lists 
(METL)  are  collective  tasks  that  combine  individual  and  leader  battle  tasks.  Tasks  within 
the  5 1C  SMCT  are  examples  of  individual  tasks  that  support  these  overall  mission- 
essential  tasks.  Individual-  and  mission-  essential  tasks  are  rated  in  FM  7-1  as  follows: 
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ASSESSMENT  RATINGS 


The  commander’s  training  assessment  is  required  for  each  METL 
task.  Task  proficiency  is  rated  as — 

“T”  (trained):  The  unit  is  trained  and  has  demonstrated  proficiency  in 
accomplishing  the  task  to  the  Army  standard.  The  leader  judges  task 
performance  to  be  free  of  significant  shortcomings.  Training  on  “T” 
tasks  is  designed  to  sustain  proficiency  on  that  task. 

“P”  (needs  practice):  The  unit  can  perform  the  task  with  some 
shortcomings.  Performance  has  demonstrated  that  the  unit  does  not 
achieve  the  standard  without  some  difficulty  or  has  failed  to  perform 
some  task  steps  to  standard.  The  shortcomings  are  not  severe  enough 
to  require  complete  retraining.  Only  refresher  training  is  required. 

“U”  (untrained):  The  unit  cannot  demonstrate  an  ability  to  achieve 
wartime  proficiency.  The  leader  prepares  a  comprehensive  plan  to 
train  all  supporting  tasks  not  executed  to  standard.  Unless  the  task  is  a 
new  METL  task,  a  rating  of  “U”  indicates  a  serious  training 
deficiency  and  reflects  on  the  unit’s  wartime  readiness  posture. 

EVALUATION  RATINGS 

Evaluation  ratings  are  given  for  specific  task  proficiency  and  should 
not  be  confused  with  leader  assessments.  Evaluation  ratings  are 
ratings  assigned  directly  to  the  perfonnance  of  a  specific  task  or 
component  steps  of  a  task.  The  standard  evaluation  ratings,  discussed 
further  in  chapter  6,  are  as  follows: 

“GO”:  The  task  or  perfonnance  step  of  a  task  was  performed  to 
standard.  A  rating  of  GO  is  nonnally  awarded  if  all  steps  in  the  task 
are  passed. 

“NO  GO”:  The  task  or  any  performance  step  in  the  task  was  not 
performed  to  standard.  (DoA,  2003,  p.  4-6) 

How  commanders  develop  a  training  plan  and  evaluate  individual  tasks  is  relevant 
to  this  project  because  this  reveals  ways  commanders  might  consider  using  the 
proficiency-assessment  test  developed  in  this  project.  It  is  not  the  research  team’s  intent 
to  develop  an  assessment  tool  for  commanders  to  automatically  determine  specific  battle- 
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task  ratings  as  defined  above.  By  understanding  this  construct,  the  research  team  was 
able  to  write  specific  and  relevant  test  instructions. 

It  is  the  hope  of  the  research  team  that  the  assessment  tool  developed  in  this 

project  will  satisfy  some  of  the  requirements  outlined  in  Army  training  doctrine  to 

establish  individual  self-development  programs.  The  following  excerpt  from  FM  7-0 

describes  the  role  of  assessment  in  establishing  individual  self-development  programs: 

Self-development  starts  with  an  assessment  of  individual  strengths, 
weaknesses,  potential,  and  developmental  needs.  Commanders  and 
leaders  provide  feedback  that  enables  subordinates  to  determine  the 
reasons  for  their  strengths  and  weaknesses.  Together,  they  prioritize 
self-development  near-term  and  long-term  goals  and  determine 
courses  of  action  to  improve  performance.  Self-development  is — 

A  planned  process  involving  the  leader  and  the  subordinate  being 
developed.  It  enhances  previously  acquired  skills,  knowledge, 
behaviors,  and  experience;  contributes  to  personal  development;  and 
highlights  the  potential  for  progressively  more  complex  and  higher- 
level  assignments.  Self-development  focuses  on  maximizing 
individual  strengths,  minimizing  weaknesses,  and  achieving 
individual  personal  and  professional  development  goals. 

Initially  very  structured  and  generally  narrow  in  focus.  The  focus 
broadens  as  individuals  understand  their  strengths  and  weaknesses, 
determine  their  individual  needs,  and  become  more  experienced.  Each 
soldier’s  knowledge  and  perspective  increases  with  experience, 
institutional  training,  and  operational  assignments,  and  is  accelerated 
and  broadened  by  specific,  goal-oriented  self-development  actions. 

(DoA,  2003,  p.  1-12) 


The  PAT  developed  during  this  research  can  be  used  by  individual  soldiers — and 
their  leaders  and  commanders — to  conduct  an  assessment  of  that  soldier’s  strengths  and 
weaknesses  in  the  SMCT  tasks.  These  assessments  can  then  be  used  to  design  follow-on 
training.  The  PAT  can  measure  the  success  of  this  follow-on  training  designed  as  a  result 
of  that  assessment. 

The  six  core  adult-leaming  principles  should  be  considered  when  establishing  a 
self-development  program  (Knowles,  Holton  &  Swanson).  Figure  4  depicts  the  six  core 
adult  learning  principles: 
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Figure  4.  Core  Adult  Learning 
(Knowles,  Holton  &  Swanson,  1998,  p.  4) 


Knowles,  Holton  and  Swanson  expand  upon  the  concept  of  the  adult-learning 
model  and  provide  eight  steps  to  guide  field  supervisors  and  subordinates  to  develop  a 
learning  contract.  These  steps  are  as  follows: 

1 .  Diagnose  your  learning  needs, 

2.  Specify  your  learning  objectives, 

3.  Specify  learning  resources  and  strategies, 

4.  Specify  evidence  of  accomplishment, 

5.  Specify  how  evidence  will  be  validated, 

6.  Review  the  contract, 

7.  Carry  out  the  contract,  and 

8.  Evaluation  of  your  learning. 
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A  learning  contract  should  be  oriented  in  such  a  way  that  the  learner  assumes  a 
major  responsibility  to  ensure  that  the  contract  is  relevant  (Piskurich,  Beckschi  &  Hall, 
2000).  Piskurich  et  al.  provides  the  following  reasons  to  justify  the  increased  use  of 
learning  contracts  to  satisfy  training  and  development  objectives: 

1.  To  provide  more  appropriate  learning  for  individuals  with  diverse 
backgrounds,  experiences,  and  learning  styles; 

2.  To  meet  the  needs  of  learners  in  specialized  areas; 

3.  To  meet  needs  of  learners  in  rapidly  changing  fields  when  no 
appropriate  curriculum  or  training  is  available; 

4.  To  meet  the  needs  of  learners  at  a  distance; 

5.  To  save  training  dollars;  and 

6.  To  develop  self-directed,  reflective,  continuing  learners  who  can 
contribute  to  the  success  and  growth  of  the  organization. 


Several  methods  and  models  are  available  for  supervisors  to  evaluate  implemented 
training  programs  (Phillups,  1997;  Piskurich,  Beckschi  &  Hall,  2000).  Assessment 
should  occur  at  every  level  of  training  development. 


This  section  of  the  literature  review  examined  how  assessment  fits  within  training 
and  leadership  development  models,  both  inside  and  out  of  the  military.  Additionally,  we 
reviewed  relevant  information  in  developing  learning  contracts  and  adult  learning 
principles.  This  infonnation  is  relevant  for  establishing  a  context  of  how  assessments  are 
used  towards  self-development  programs  as  well  as  for  serving  as  a  possible  guide  for  the 
research  team  during  the  development  and  publication  of  the  commander’s  assessment- 
tool  guide. 


The  literature  review  until  now  has  served  in  providing  the  research  team  with  the 
historical  background  of  contingency-contracting  training  and  where  a  test  would  fit  in  a 
training  methodology.  The  following  sections  of  the  literature  discuss  the  core  test 
principles,  test  construction  methods,  test  blueprints  and  item  writing.  These  sections 
deal  directly  with  the  mechanics  of  writing  the  test.  This  section  provided  the  research 
team  with  the  knowledge  of  how  to  develop  and  write  the  questions  for  the  PAT. 
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2.  Core  Test  Principles 

Tests  can  be  categorized  into  numerous  types.  Examples  of  these  categories  are 
intelligence,  ability,  personality,  objective,  projective,  and  attitude,  among  others  (Kline, 
1986).  Tests  are  specific  tools  designed  and  administered  to  serve  a  specific  function. 
These  functions  can  be  categorized  into  three  areas:  prediction,  assessment,  and  trait 
measurement  (Nunnally,  1972).  Tests  designed  to  a  predictive  function  attempt  to 
forecast  how  an  individual  will  behave  in  a  particular  situation.  An  example  of  this  test 
might  be  a  college  placement  exam.  Assessment  tests  do  not  attempt  to  predict  future 
behavior,  but  rather  directly  measure  the  current  performance  of  an  individual  at  a 
specific  point  in  time.  The  level  of  performance  measured  depends  upon  the  type  of  test 
developed  and  administered.  Trait-measurement  tests  are  designed  to  measure  specific 
psychological  traits  such  as  anxiety  or  depression. 

Another  way  to  categorize  tests  is  to  classify  them  as  either  a  criterion-referenced 
test  or  a  nonn-referenced  test  (Marshall  &  Hales,  1972).  Criterion-referenced  tests 
attempt  to  determine  if  an  individual  has  mastered  a  subject  area  in  a  specific  learning 
objective.  This  measurement  could  be  interpreted  in  a  variety  of  ways,  such  as  deciding 
if  an  individual  is  ready  to  move  to  another  subject  or  higher  level  of  mastery.  This  type 
of  test  is  often  used  as  a  diagnostic  instrument.  Norm-referenced  tests  attempt  to  measure 
an  individual  against  a  referenced  norm,  such  as  a  group  of  colleagues  or  other 
professionals.  This  type  of  testing  attempts  to  rank  students  on  a  scale  of  magnitude 
within  the  distribution  of  a  specific  population.  The  proficiency-assessment  test 
developed  within  the  scope  of  this  project  can  be  classified  as  a  criterion-referenced  test. 

Tests  involve  three  key  players:  the  test  user,  test  writer,  and  test  taker.  The 
Standards  for  Educational  and  Psychological  Tests  defines  a  test  user  as  one  who 
chooses  tests  to  give,  interprets  scores,  or  makes  decisions  based  on  test  scores. 
Dependent  upon  the  use  or  function  of  the  test,  the  following  standards  are  considered 
essential  for  test  users  (APA,  1974,  p.  58-65): 

1 .  Test  user  should  have  a  general  knowledge  of  measurement 
principles  and  of  the  limitations  of  test  interpretations,  and 
how  well  they  match  the  qualifications  required  to  use  a 
specific  test. 
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2.  A  test  user  should  know  and  understand  the  literature 
relevant  to  the  test  she  uses. 

3.  One  who  has  the  responsibility  for  decisions  about 
individuals  or  policies  that  are  based  on  test  results  should 
have  an  understanding  of  psychological  or  educational 
measurement. 

4.  Principal  test  users  within  an  organization  should  make 
every  effort  to  be  sure  that  all  those  in  the  organization  who 
are  charged  with  responsibilities  related  to  test  use  and 
interpretation  have  received  training  appropriate  to  those 
responsibilities. 

5.  Tests  users  should  seek  to  avoid  bias  in  test  selection  and 
interpretation  or  even  the  appearance  of  discriminatory 
practice. 

6.  Institutional  test  users  should  establish  procedures  for 
periodic  internal  review  of  test  use. 

7.  The  choice  or  development  of  tests  or  other  assessment 
procedures  should  be  based  on  clearly  formulated  goals  and 
hypothesis. 

8.  A  test  user  should  consider  more  than  one  variable  for 
assessment  and  the  assessment  of  any  given  variable  by 
more  than  one  method. 

9.  In  choosing  an  existing  test,  a  test  user  should  relate  its 
history  of  research  and  development  to  his  intended  use  of 
the  instrument. 

10.  A  test  user  is  expected  to  follow  carefully  the  procedures 
described  in  the  test  manual. 

1 1 .  The  test  administrator  is  responsible  for  establishing 
conditions,  consistent  with  the  principle  of  standardization, 
that  enable  each  examinee  to  do  his  or  her  best. 

12.  The  test  user  shares  with  the  test  developer  or  distributor  a 
responsibility  for  maintaining  test  security. 

13.  A  test  score  should  be  interpreted  as  an  estimate  of 
performance  under  a  given  set  of  circumstances.  It  should 
not  be  interpreted  as  some  absolute  characteristic  of  the 
examinee  or  as  something  permanent. 


Anastasi  (1982)  recommends  that  two  main  reasons  to  control  the  use  of  a  test  is  to 
ensure  that  it  is  administered  by  a  qualified  examiner  and  to  protect  the  test  content  from 
being  compromised,  which  would  invalidate  the  assessment.  A  qualified  examiner  must 
essentially  know  how  to  select  the  right  test,  administer  the  test,  and  properly  interpret  the 
results. 
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Every  text  on  measurement  theory  and  test  construction  that  the  research  team 
reviewed,  listed  reliability  and  validity  as  two  essential  test  characteristics.  Some 
references  added  additional  characteristics  such  as  objectivity,  balance,  fairness,  and 
practicality  to  the  list  (Marshall  &  Hales,  1972).  Another  author  considers  efficiency  as  a 
characteristic  and  highlights  current  theories  and  models  designed  to  improve  test 
efficiency  (Brennan,  2006).  For  simplicity,  this  literature  review  focuses  on  reliability 
and  validity  as  the  two  main  test  characteristics. 

Reliability,  in  essence,  measures  the  consistency  of  a  test.  The  more  consistent  a 
test  is,  the  higher  the  probability  the  same  individual  will  obtain  a  similar  score  when 
retesting  on  an  identical  test  or  equivalent  version  (Anastasi,  1982).  Statistically,  there 
are  multiple  methods  to  derive  reliability: 

Reliability  can  be  defined  in  several  ways — such  as  the  proportion  of 
observed-score  variance  that  is  true-score  variance,  the  squared 
correlation  between  true  scores  and  observed  scores,  or  the  correlation 
between  observed  scores  on  two  parallel  tests.  Several  estimates  of 
reliability  are  widely  used,  including  test/retest,  parallel-forms, 
alternate  fonns,  and  internal-consistency  estimates.  (Allen  &  Yen, 

2001,  p.  91) 

Test  developers  want  to  minimize  the  effects  of  chance  on  reliability  (Wood,  1960). 
Wood  identifies  the  following  three  sources  of  chance  variation: 

1 .  Scorer  unreliability  occurs  when  the  scorer  through  subjectivity  or  bias  induces 
variability  into  an  assessment  or  testing  process. 

2.  Content  unreliability  occurs  from  a  poor  sampling  of  content.  It  typically  occurs 
where  a  score  of  an  individual  depends  heavily  on  the  sample  of  questions 
selected  from  a  test  speculation. 

3.  Temporal  unreliability  includes  a  variety  of  factors  that  would  explain  how  an 
individual’s  score  varied  at  different  times.  One  example  of  this  is  the  varying 
ability  of  an  individual  to  remember  responses  from  the  last  test  when  taking  the 
second  test. 

According  to  Kline  (1996),  other  recommended  sources  of  unreliability  include  guessing, 
test  length,  and  poor  test  instructions,  or  simply  a  sick  or  ill  test  subject. 
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Validity  is  how  well  a  test  meets  its  designed  function.  A  test  might  be  extremely 
reliable,  yet  at  the  same  time,  be  invalid.  It  is  never  acceptable  to  accept  a  test  as  valid 
until  something  proves  it  as  such  (Nunnally,  1972): 

Validity  can  be  assessed  in  several  ways,  depending  on  the  test  and  its 
intended  use.  The  three  major  types  of  validity  are  content  validity, 
criterion-related  validity  and  construct  validity.  Detenninations  of 
criterion-related  validity  and  construct  validity  involve  the  calculation 
and  examination  of  correlations  or  other  statistics.  Content  validity, 
however,  does  not  involve  any  statistical  calculations.  (Allen  &  Yen, 

2001,  p.  95) 

Content  validity  is  achieved  through  an  individual,  subjective  judgment. 
Essentially,  it  is  established  when  someone  examines  the  test  and  determines  it  correctly 
measures  what  it  was  designed  to  measure.  This  type  of  validity  is  more  subject  to  error 
and  should  not  be  relied  upon  as  the  sole  detennination  of  a  test’s  validity  (Allen  &  Yen, 
2001).  The  research  team  recognizes  the  limitations  of  content  validity,  but  we  consider 
it  the  most  achievable  type  of  validity  within  the  scope  of  the  project  and  a  primary 
concern  during  test  construction.  Nunnally  &  Bernstein  (1978)  discuss  content  validity 
in  the  following  way: 

Rather  than  test  the  validity  of  measures  after  they  are  constructed, 
one  should  ensure  validity  by  the  plan  and  procedures  of  construction 
[...].  The  validity  of  the  measure  is  judged  by  the  character  of  the 
plan  and  by  the  apparent  skill  with  which  the  plan  has  been  carried 
out.  If  it  is  agreed  by  most  potential  users  of  the  test,  or  at  least  by 
persons  in  positions  of  responsibility,  that  the  plan  was  sound  and 
well  carried  out,  the  test  has  a  high  degree  of  content  validity. 

The  simple  example  above  illustrates  the  two  major  standards  for 
ensuring  content  validity:  (1)  a  representative  collection  of  items  and 
(2)  “sensible”  methods  of  test  construction,  (pp.  80-81) 

Criterion-related  validity  is  derived  by  establishing  a  correlation  between  a  test 
score  and  a  specific  criterion’s  score.  Criterion-related  validity  is  interchangeable  with 
the  term  predictive  validity  (Nunnally  &  Bernstein,  1978).  Brenan  (2006)  describes  the 
evolution  of  all  three  models,  the  limitations  of  each  model,  and  the  methods  of 
estimation.  It  is  not  the  research  team’s  intention  to  describe  the  science  of  validity  in 


41 


detail,  but  rather  to  identify  it  as  a  core  test  principle  and  recognize  its  relevance  towards 
the  assessment  tool  developed  in  this  project. 

The  Standards  for  Educational  and  Psychological  Tests  provides  essential 
standards  for  test  developers  to  achieve  both  reliability  and  validity.  Many  of  these 
standards  are  achieved  throughout  the  entire  test-construction  process.  Steps  are  taken  by 
test  creators  to  reevaluate  validity  and  reliability  using  test  trials  and  statistical  analysis 
before  a  test  is  published  in  its  final  form. 

3.  Test  Construction 

This  section  of  the  literature  review  examines  recognized  steps  of  test 
construction  and  identifies  rules  and  procedures  for  some  of  the  key  steps.  Allen  &  Yen 
(2001)  identify  the  following  basic  steps  in  test  development: 

1 .  Plan  the  test.  During  this  step,  developers  establish  a  test  map  or  test 
blueprint  to  document  what  areas  the  test  items  will  cover. 

2.  Write  the  test  items  or  questions. 

3.  Administer  all  written  test  items  to  a  sample  population.  Generally,  a 
minimum  of  50  persons  is  reasonable,  but  several  hundred  is  optimal. 
The  sample  population  should  represent  the  population  for  which  the 
test  was  developed. 

4.  Conduct  item  analysis  using  the  results  of  the  sample.  During  this 
step,  test  developers  refine  test  items  and  eliminate  those  that  are 
invalid. 

5.  Administer  the  revised  tests  to  another  sample  population  and  continue 
steps  2-4  until  a  satisfactory  test  is  produced. 

Downing  and  Haladyna  (2006),  list  twelve  steps  for  effective  test  development: 

1 .  Overall  plan, 

2.  Content  definition, 

3.  Test  specifications, 

4.  Item  development, 

5.  Test  design  and  assembly, 

6.  Test  production, 

7.  Test  administration, 

8.  Scoring  test  responses, 

9.  Passing  scores, 

10.  Reporting  test  results, 

1 1 .  Item  banking,  and 

12.  Test  technical  report,  (p.  5) 
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Building  a  test  blueprint  and  item  writing  are  two  of  the  common  steps  found  between 
these  test  development  models  that  we  intend  to  execute  within  the  scope  of  this  project. 
What  follows  in  this  literature  review  is  a  more  detailed  look  at  these  steps,  and  in 
Chapter  IV  of  this  project,  we  identify  what  procedures  we  followed  to  develop  the 
assessment  tool. 

4.  Test  Blueprints 

Test  blueprints  act  as  a  road  map  for  test  construction  in  laying  out  which  subject 
areas  the  test  developer  wants  to  test  and  at  what  cognitive  domain  the  subject  area 
should  be  tested. 

Test  specifications  are  often  called  blueprints  because  they  specify 
how  the  test  or  pool  is  to  be  constructed.  [...]  Specifications  for  a 
test  fonn  or  pool  describe  its  content,  form,  and  functional 
requirements.  (Brennan,  2006,  p.  315) 

At  a  minimum,  test  blueprints  or  specifications  should  contain  the  following  information 
(Downing  &  Haladyna,  2006): 

1 .  The  type  of  testing  format  to  be  used  (selected  response  versus  constructed 
response), 

2.  The  total  number  of  test  items  as  well  as  the  type  or  fonnat  of  test  items, 

3.  The  cognitive  classification  system  to  be  used, 

4.  Whether  or  not  the  test  items  or  performance  prompts  will  contain  visual 
stimuli, 

5.  The  expected  item  scoring  rules, 

6.  How  test  scores  will  be  interpreted  (norm  or  criterion-referenced),  and 

7.  The  time  limit  for  each  item.  (p.  9) 


Subject-matter  experts  and  administrative  officials  should  review  test  blueprints  prior  to 
implementation  (Nunnally  &  Bernstein,  1978). 

Several  texts  recognized  the  use  of  Bloom’s  Taxonomy  of  Educational  Objectives 
as  a  common  method  to  classify  cognitive  domains  within  a  test  blueprint  (Downing& 
Haladyna,  2006;  Brennan,  2006;  Marshall  &  Hales,  1972;  Kline,  1986).  (Brennan,  2006) 
(Kline,  1986)  The  research  team  uses  this  taxonomy  during  the  test-specification  step,  so 
a  brief  summary  of  Bloom’s  Taxonomy  is  included  here. 
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Bloom  developed  a  taxonomy  of  educational  objectives  in  order  to  assist  teachers, 
professionals,  and  researchers  in  categorizing  educational  goals  and  to  guide  them  with 
curricular  and  evaluation  problems  (Bloom,  1956).  His  work  divided  the  cognitive 
domain  into  six  categories:  knowledge,  comprehension,  application,  analysis,  synthesis, 
and  evaluation.  He  provided  examples  of  illustrative  test  items  within  each  domain.  His 
classic  work  was  revised  by  several  of  his  pupils  (Anderson  et  ah,  2002).  Their  revised 
version  of  Bloom’s  taxonomy  is  depicted  in  Figure  5  and  6.  Anderson  et  al.  establishes  a 
knowledge  dimension  that  is  divided  into  four  categories — factual,  conceptual, 
procedural,  and  meta-cognitive — and  suggests  using  a  taxonomy  table  as  an  assessment- 
development  tool.  Other  authors  demonstrate  the  use  of  similar  matrices  to  develop 
blueprints  within  a  taxonomy  construct  (Marshall  &  Hales,  1972;  Downing  &  Haladyna, 
2006). 
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3.1  EXECUTING 


Carrying  out  Applying  a  procedure  to  a  familiar  task  (e.g.,  Divide  one 

whole  number  bv  another  whole  number,  both  with 


3.2  implementing  Using 


CATEGORIES 

&  Cognitive  Alternative 

PROCESSES  NAMES 

DEFINITIONS  AND  EXAMPLES 

1 .  remember — Retrieve  relevant  knowledge  from  long  term  memory 

1 ,-1 

RECOGNIZING 

Identifying 

Locating  knowledge  in  long-term  memory  that  is  consistent 
with  presented  material  (e.g..  Recognize  the  dates  of 
important  events  in  U.S.  history) 

1.2 

RECALLING 

Retrieving 

Retrieving  relevant  knowledge  Irom  long-term  memory 
(e.g.,  Recall  the  dates  of  important  events  in  U.S.  history) 

J -  ' 

2 .  understand — Construct  meaning  from  instructional  messages,  including  oral,  WTitten,  and 

graphic  communication  _ _ _ 

.  2.1 

INTERPRETING 

Clarifying, 

paraphrasing, 

representing, 

translating 

Changing  from  one  form  of  representation  (e.g.,  numerical) 
to  another  (e.g.,  verbal)  (e.g..  Paraphrase  important  speeches 
and  documents) 

I  2.2 

EXEMPLIFYING 

Illustrating, 

instantiating 

Finding  a  specific  example  or  illustration  of  a  concept  or  prin¬ 
ciple  (e.g..  Give  examples  of  various  artistic  painting  styles) 

2.3 

1 

CLASSIFYING 

Categorizing, 

subsuming 

Determining  that  something  belongs  to  a  category  (e.g., 
concept  or  principle)  (e.g..  Classify  observed  or  described 
cases  of  mental  disorders) 
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SUMMARIZING 

Abstracting, 

generalizing 

Abstracting  a  general  theme  or  major  point(s)  (e.g.,  Write  a 
short  summary  of  the  events  portrayed  on  a  videotape) 

2.S 

INFERRING 

Concluding, 

extrapolating, 

interpolating, 

predicting 

Drawing  a  logical  conclusion  from  presented  information 
(e.g.,  In  learning  a  foreign  language,  infer  grammatical 
principles  from  examples) 

2.6 

Comparing 

Contrasting. 

mapping. 

Detecting  correspondences  between  two  ideas,  objects,  and 
the  like  (e.g..  Compare  historical  events  to  contemporary 

1 

matching 

situations) 

2.7 

Explaining 

Constructing 

models 

Constructing  a  cause-and-effect  model  of  a  system  (e.g..  Ex¬ 
plain  the  causes  of  important  18th-century  events  in  France) 

3. 

apply — Carry  out  or  use  a  procedure  in  a  given  situation 

_ _ . _ _ —  —  —  1  i 

multiple  digits) 

Applying  a  procedure  to  an  unfamiliar  task  (e.g..  Use  New¬ 
ton's  Second  Law  in  situations  in  which  it  is  appropriate) 


Figure  5.  Revised  Bloom’s  Taxonomy  of  Learning  Objectives 

(Anderson  et  al.,  2002,  p.  66) 
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Categories 

&  COGNITIVE 

PROCESSES 

Alternative 

Names 

Definitions  and  Examples 

4.  analyze — Break  material  into  its  constituent  parts  and  determine  how  the  parts  relate  to  one 
another  and  to  an  overall  structure  or  purpose 

4.1 

Differentiating  Discriminating, 
distinguishing, 
focusing, 
selecting 

Distinguishing  relevant  from  irrelevant  parts  or  impor-  j 
tanf  from  unimportant  parts  of  presented  material 
(e  g..  Distinguish  between  relevant  and  irrelevant 
numbers  in  a  mathematical  word  problem) 

4.2 

ORGANIZING 

Finding 

coherence, 

intergrating, 

outlining, 

parsing, 

structuring 

Determining  how  elements  fit  or  function  within  a 
structure  (e.g.,  Structure  evidence  in  a  historical 
description  into  evidence  for  and  against  a  particular 
historical  explanation) 

4.3 

Attributing 

Deconstructing 

Determine  a  point  of  view,  bias,  values,  or  intent  under¬ 
lying  presented  material  (e.g..  Determine  the  point  of 
view  of  the  author  of  an  essay  in  terms  of  his  or  her 
political  perspective) 

3 .  e va  L u  at E— Make  judgments  based  on 

criteria  and  standards 

5.1 

Checking 

Coordinating, 

detecting, 

monitoring, 

testing 

Detecting  inconsistencies  or  fallacies  within  a  process  or 
product;  determining  whether  a  process  or  product  has 
internal  consistency;  detecting  tire  effectiveness  of  a  pro¬ 
cedure  as  it  is  bemg  implemented  (e.g.,  Determine  if  a 
scientist's  conclusions  follow  from  observed  data) 

3.2 

Critiquing 

Judging 

Detecting  inconsistencies  between  a  product  and  exter¬ 
nal  criteria,  determining  whether  a  product  has  exter¬ 
nal  consistency;  detecting  the  appropriateness  of  a  pro¬ 
cedure  for  a  given  problem  (e.g,.  Judge  which  of  two 
methods  is  the  best  way  to  solve  a  given  problem) 

6.  create — Pul  elements  together  to  form  a  coherent  or  functional  whole,  reorganize  elements 
into  a  new  pattern  or  structure 

6.1 

Generating 

Hypothesizing 

Coming  up  with  alternative  hypotheses  based  on 
criteria  (e.g.,  Generate  hypotheses  to  account  for  an 
observed  phenomenon) 

6.2 

Planning 

Designing 

Devising  a  procedure  for  accomplishing  some  task  (e.g.. 
Plan  a  research  paper  on  a  given  historical  topic) 

6.3 

Producing 

Constructing 

Inventing  a  product  (e.g.,  Build  habitats  for  a  specific 
purpose) 

Figure  6.  Revised  Bloom’s  Taxonomy  of  Learning  Objectives 
(Anderson  et  al,  2002,  p.  67) 


5.  Item  Writing 

Items,  or  rather  questions,  can  be  divided  into  two  general  formats  (Osterlind, 
1997).  Selected-response  items  occur  when  the  examinee  is  given  both  the  problem  and 
the  correct  solution.  The  examinee  must  then  choose  the  correct  solution  over  other 
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incorrect  solutions  provided  as  options.  Figure  7  depicts  an  example  of  this  type  of 
format  and  gives  the  reader  an  idea  of  test- item  nomenclature. 


Use  me  Australian  and  road  the  passage  below  to  answer  Direction 
the  question.  — 1 


Graphic 


Charley  did  an  experiment  in  which  he  floated  an 
eyedropper  in  a  jar  of  water,  He  left  the  jar  on  a  shelt  tor 
one  hour.  Then  he  recorded  the  level  of  water  in  the 
eyedropper,  Next,  he  heated  the  jar  containing  the 
eyedropper  for  30  seconds. 

Which  variable  dkl  Charley  change  in  this  experiment? 


Taxi 


] 


Stem 


A.  kind  of  eyedropper 

B.  level  of  water  m  the  iar 

C.  amount  of  lime  between  temperature  measurements 
•D  temperature  of  the  air  and  water  in  the  jar 


Distractors 

Correct 

Response 


Figure  7.  Example  of  Selected-response  Item  and  Test-item  Nomenclature 

(Osterlind,  1997,  p.  34) 


Constructive-response  style  items  only  provide  the  examinee  the  problem.  The 
examinee  is  required  to  respond  by  providing  the  perceived  correct  response.  Each 
format  has  its  advantages  and  disadvantages.  For  example,  it  is  typically  easier  and  less 
resource  intensive  to  score  selected-response  items.  Items  can  further  be  refined  into 
specific  formats  such  as  multiple  choice,  true/false,  matching,  sentence  completion,  short 
answer,  essay,  and  so  forth. 

There  are  two  types  of  test- item  writers:  professional  and  casual  (Osterlind, 
1997).  As  casual  writers,  we  used  collected  checklists  to  guide  us  during  the  item¬ 
writing  step.  Many  of  the  references  we  reviewed  contained  useful  item-writing  tips. 
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These  tips  not  only  focused  on  the  specifics  of  writing  one  test  item  but  also  included 
more  expansive  topics,  such  as  how  to  order  items  within  a  test  and  ethical  or  legal 
concerns  for  using  copyrighted  test  items. 

Faulty  test  items  are  typically  ambiguous  and  trivial  (Nunnally  &  Bernstein, 
1978).  Item  writers  should  establish  a  random  system  to  position  the  correct  answer 
among  alternatives  in  order  to  mitigate  biases  within  the  test  (Wood,  1960).  Five  rules  in 
writing  good  multiple-choice  items  include  (Kline,  1986)  the  following: 

1 .  Simplicity  (in  writing), 

2.  All  distracters  should  be  capable  of  distracting  subjects, 

3.  Only  one  option  should  be  correct, 

4.  The  answer  to  one  question  should  not  give  clues  to  the  answer  to  another, 
and 

5.  Avoid  testing  the  trivial  because  it  is  easy  to  test. 

Kline  (1986)  further  suggests  that  one  technique  for  determining  which  item 
format  best  suits  a  particular  subject  is  to  write  them  all.  Through  this  process,  a  test 
writer  can  eliminate  those  formats  that  do  not  suit  a  particular  subject  and  then  select 
from  the  remaining  options  the  fonnat  that  best  achieves  the  intended  purpose  of  the 
item.  A  more  exhaustive  list  of  31  general  item- writing  guidelines  can  be  found  in 
Developing  and  Validating  Multiple-Choice  Test  Items.  Some  of  these  guidelines  include 
the  following:  avoid  using  “all  of  the  above,”  avoid  trick  items,  make  the  stem  as  brief  as 
possible,  place  the  main  idea  of  the  item  in  the  stem  and  not  in  the  choices,  and  so  forth 
(Haladyna,  2004).  Chapter  IV,  Methodology,  clearly  describes  which  rules  we  used  to 
guide  us  through  item  writing. 

D.  SUMMARY 

This  chapter  reviews  related  studies,  reports,  articles  and  other  literature  on  the 
topics  of  contingency  contracting  and  test  construction  and  evaluation.  Within  the  realm 
of  contingency  contracting,  the  research  team  focused  on  the  history  and  evolution  of 
tasks  related  to  training  CCOs  for  deployment  to  a  contingency  environment.  The  review 
examined  literature  pertaining  to  contingency  contracting  on  all  the  major  conflicts  the 
United  States  military  has  taken  part  in,  training  methods  and  tasks  for  contingency 
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contractors,  and  the  problems  plaguing  contingency  contracting,  as  highlighted  by  the 
Gansler  report.  Our  review  of  contingency  contracting  culminated  with  a  discussion  of 
the  establishment  of  the  ECC  and  the  development  of  the  SMCT.  Within  our  discussion 
of  test  construction  and  evaluation,  the  research  team  focused  on  overarching  principles 
of  assessment  as  a  subset  of  the  adult-learning  model,  core  test  principles,  test 
construction  and  test  blueprints,  Bloom’s  Taxonomy,  and  item  writing,  as  it  relates  to  test 
construction.  The  SMCT,  developed  to  provide  CCOs  and  their  leaders  with  a  standard 
by  which  to  train,  lacks  a  standardized  test  for  measuring  performance  against  that 
standard.  The  research  team  developed  a  test  by  combining  these  two  distinct  disciplines 
using  knowledge  gained  during  this  literature  review. 

In  conducting  this  literature  review,  we  realize  that  there  are  certain  limitations  on 
our  ability  to  develop  a  comprehensive  assessment  test  for  CCOs.  The  most  obvious  is 
that  we  are  not  professional  test  developers.  The  field  of  test  construction  and  evaluation 
is  extremely  broad  and  quite  extensive.  Professional  test  developers  could  rightfully  take 
years  to  develop  an  assessment  test  for  CCOs,  while  this  research  has  to  be  completed 
within  certain  time  constraints  to  satisfy  academic  requirements.  Given  our  limitations  in 
both  experience  as  test  developers  and  in  the  time  constraints  of  the  academic  schedule,  a 
modified  test-development  plan  was  developed  that  applies  the  principles  of  test 
construction  and  evaluation.  Chapter  IV,  Methodology,  covers  in  detail  the  actual  steps 
we  took  in  developing  the  PAT  and  how  it  varies  from  the  theoretical  test-construction 
principles  discussed  in  this  chapter. 

The  next  chapter  focuses  on  frameworks  of  the  contingency-contracting 
environment.  The  research  team  focuses  primarily  on  the  historical  context  and 
establishment  of  the  U.S.  Army’s  Expeditionary  Contracting  Command  (ECC).  The 
chapter  further  discusses  the  current  MOS  5 1C  development  and  training  model  and  how 
the  Soldier ’s  Manual  of  Common  Tasks  relates  to  this  model. 
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III.  CONTINGENCY  CONTRACTING  FRAMEWORK— THE  ECC 

AND  THE  SMCT 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  an  overview  of  the  ECC  as  an 
organization  and  to  describe  in  detail  the  content  of  the  SMCT.  In  this  chapter,  we 
discuss  the  context  under  which  the  ECC  was  established  and  describe  the  ECC 
organizational  structure  and  training  methodology.  We  also  describe  how  the  SMCT  fits 
in  to  the  ECC’s  training  strategy  and  breakdown  the  tasks  in  the  document  by  their 
subcomponent  parts. 

B.  ESTABLISHMENT  OF  THE  ECC 

The  U.S.  Army’s  Expeditionary  Contracting  Command  (ECC)  was  established  in 
January  2008  in  direct  response  to  the  Gansler  Commission’s  recommendations  to 
prepare,  train  and  provide  skilled  contracting  personnel  in  support  of  U.S.  Army 
contingency  operations  (DoA,  2009a).  Among  the  Gansler  Commission’s  four 
overarching  recommendations,  the  first  two  addressed  the  need  for  changes  in 
institutional  and  organizational  structure  to  fix  many  of  the  Army’s  failures  identified  in 
the  report.  The  first  one  recommends  an  increase  in  personnel,  and  the  second, 
recommends  the  establishment  of  the  parent  organization  of  those  personnel. 
Specifically,  recommendation  number  two,  restructure  organization  and  restore 
responsibility  to  facilitate  contracting  and  contract  management  in  expeditionary  and 
CONUS  operations,  lists  several  sub-recommendations.  Among  these  are  the 
recommendation  to  establish  the  Army  Contracting  Command  (ACC)  and  the 
Expeditionary  Contracting  Command  (ECC)  as  a  subordinate  element  of  the  ACC 
(Gansler,  2007).  Figure  8  depicts  the  ACC’s  and  ECC’s  organizational  structure. 
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Figure  8.  ACC  and  ECC  Organizational  Structure 
(ECC,  2009,  October  14) 

Recommendation  number  one,  increase  the  stature,  quantity,  and  career 
development  of  the  Army’s  contracting  personnel,  both  military  and  civilian  (especially 
for  expeditionary  operations),  recommends  the  authorization  for  general-officer  billets  to 
lead  those  organizations,  the  establishment  of  MOS  5 1C,  and  increasing  the  Army’s 
contracting  workforce  by  400  uniformed  personnel  and  1,000  civilian  personnel  (Gansler, 
2007).  It  is  these  increases  in  unifonned  personnel  will  eventually  fill  the  ranks  of  this 
new  organization.  While  the  Gansler  Commission  provided  the  recommendations  for 
these  new  changes,  General  Order  (GO)  No.  2009-20,  published  July  8,  2009,  officially 
established  the  ACC  and  ECC,  effective  January  30,  2008  (DoA,  2009,  July  8). 
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C.  ECC  ORGANIZATIONAL  STRUCTURE 

The  ECC  is  comprised  of  a  command  headquarters  and  six  contingency  support 
brigades  (CSBs).  These  CSBs  are  aligned  with  each  Army  geographical  theater.  Each 
CSB  is  comprised  of  one  or  more  contingency-contracting  battalion  (CCBn),  which,  in 
turn,  generally  contains  two  or  more  contingency  contracting  teams  of  four  MOS  5 1C 
personnel.  The  ECC  headquarters  provides  the  command  and  control  of  the  Army’s 
expeditionary-contracting  assets.  It  is  responsible  for  training,  equipping,  and  providing 
trained  contracting  capability  to  support  any  contingency  (ECC,  2009,  October  14).  It  is 
under  this  charter  that  the  ECC  has  developed  the  SMCT,  with  the  intent  of  establishing  a 
measurable  standard  for  CCOs  to  train  against  in  preparation  for  deployment.  The  lack 
of  an  assessment  test  to  measure  against  that  standard  necessitates  the  development  of  the 
PAT  and  is  the  main  driver  for  this  research  project. 

D.  TRAINING  METHODOLOGY 

In  developing  a  training  methodology  for  CCOs  in  this  new  organization,  the 
ECC  acknowledges  the  multitude  of  requirements  of  a  newly  assigned  CCO.  Figure  9 
depicts  the  ECC’s  CCO  Development  Model,  which  shows  the  multitude  of  requirements 
CCOs  face  to  become  proficient.  This  model  shows  the  SMCT  as  one  of  the  many 
educational  and  professional-development  requirements  of  a  CCO.  At  the  core  of  a 
CCO’s  training  is  his  warrior-task  training.  CCOs  must  remain  proficient  in  the 
warfighting  tasks  of  shooting,  moving,  and  communicating  in  order  to  survive  and  win  on 
the  battlefield.  Additionally,  a  CCO  must  take  part  in  mandatory  military  training,  such 
as  training  exercises  at  the  training  centers.  As  an  acquisition  professional,  the  CCO 
must  meet  civilian  educational  requirements  as  well  as  DAU  certification  requirements  to 
remain  proficient  and  current  in  the  acquisition  workforce.  The  36  tasks  of  the  SMCT 
serve  to  train  the  CCO  in  practical  application  of  his  craft  so  that  he  may  be  able  to 
perform  his  mission  while  deployed.  Finally,  the  contracting  tasks  the  CCO  perfonns 
while  deployed  round  out  the  multitude  of  requirements  a  CCO  must  accomplish  to  be 
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proficient.  The  proficiency-assessment  test  developed  in  this  research  will  serve  as  a 
complement  to  the  SMCT  in  this  model,  providing  feedback  to  the  CCO  and  his  leaders 
on  his  ability  to  accomplish  the  tasks. 


Warrior  Task  Training 


36  Contracting  Tasks 
In  Contracting  Offices 
per  the  SMCTs 


Balance  Many 
Requirements 


Contracting  Tasks 
Performed  During 
Contingency 


Civilian 

Education 


Mandatory  Military 
Training 


Figure  9.  CCO  Development  Model 
(ECC,  2009,  September  22) 

CCOs  assigned  to  the  ECC  may  come  from  a  multitude  of  acquisition  or 
operational  Army  experiences.  They  all,  however,  must  meet  the  minimum  education 
and  training  requirements  prior  to  being  assigned  to  the  ECC  as  a  CCO  and  must 
continue  to  train  in  preparation  for  deployment.  The  increased  contingency-contracting 
requirements  and  the  growing  number  of  CCOs  within  the  ECC,  a  relatively  new 
organization,  means  there  are  shortages  of  CCOs  deployed  in  support  of  contingency 
operations.  This  assures  that  almost  every  newly  assigned  CCO  arriving  to  a  subordinate 
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unit  within  the  ECC  will  have  an  opportunity  to  deploy.  Figure  10,  from  the  ECC’s 
briefing  on  phased  development,  depicts  the  typical  timeline  that  CCOs  currently 
experience  from  arrival  to  their  unit  through  their  first  deployment  and  beyond.  After 
being  assessed  into  the  Army  Acquisition  Corps  (AAC)  as  a  5 1C,  the  CCO  receives  the 
preliminary  functional  training,  awarding  their  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  Contracting  Level  I.  Upon  arrival  to  a  team,  the  CCO 
continues  to  train  for  deployment,  under  the  direction  of  a  team  leader  or  battalion 
commander,  on  the  various  requirements  listed  in  Figure  9.  It  is  during  this  time  that  a 
CCO  may  first  take  the  PAT  so  that  he  or  his  leaders  can  assess  his  proficiency  in  the  36 
tasks.  From  here,  the  team  leader  or  battalion  commander  can  work  with  the  CCO  to 
develop  a  specialized  training  program  to  improve  his  proficiency.  After  several  months 
of  training,  the  CCO  may  take  the  PAT  again  to  assess  proficiency  once  more. 

These  follow-on  PATs  may  be  used  to  gauge  the  CCO’s  progress  and  the 
effectiveness  of  a  commander’s  training  program.  After  a  minimum  of  one  year  of 
training  at  their  duty  station,  the  CCO  may  deploy  in  support  of  a  contingency  operation. 
One  of  the  recommendations  of  the  Gansler  Commission  that  was  adopted  by  the  Army 
was  to  not  send  an  untrained  CCO  into  theater  (Gansler,  2007).  This  standard  timeframe 
of  allowing  a  CCO  to  train  for  up  to  one  year  prior  to  deployment  was  adapted  by  the 
Army  and  the  ECC  and  is  depicted  in  Figure  10. 
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T  Continues 
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DAULVI 
Assessed  Functional 

Training  Minimum 


•  NCO  will  need  DAU  LV II  Training  and  Army  Acq. 
Intermediate  Contracting  Course  (ANCOC) 

•  MSG  goes  to  Intermediate  Qualification  Course 

•  Officer  may  need  Intermediate  Level  Education 

►NCOs  Anny  Acquisition  Basic  Course  (BNCOC)  or  Ready  Airman  Course 


Figure  10.  CCO  Development  Timeline 
(ECC,  2009,  September  22) 


E.  CONTRACT  MANAGEMENT  OVERVIEW 

The  contract-management  process  from  the  buyer’s  perspective  can  be  broken  up 
into  six  steps.  Garrett  (2007)  identifies  these  phases  as  (1)  Procurement  Planning,  (2) 
Solicitation  Planning,  (3)  Solicitation,  (4)  Source  Selection,  (5)  Contract  Administration, 
and  (6)  Contract  Closeout  or  Termination.  Rendon  &  Snider  (2008)  describe  each  of  the 
six  steps  in  the  contract-management  process  as  follows:  Procurement  Planning  involves 
identifying  what  requirement  can  be  met  by  procuring  products  or  services  from  outside 
the  organization.  This  phase  involves  whether  to  procure,  what  and  how  much  to 
procure,  and  when  to  procure  it.  Solicitation  Planning  involves  the  preparation  of  the 
documents  needed  to  support  the  solicitation.  Solicitation  is  the  process  of  actually 
obtaining  the  bids  and  proposals  from  the  prospective  sellers.  Source  Selection  is  the 
process  of  receiving  and  evaluating  the  bids  and  proposals  from  prospective  offerors  and 
determining  which  offeror  to  select.  Contract  Administration  is  the  process  of  ensuring 
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that  each  of  the  parties  involved  in  the  contract  meet  their  respective  contractual 
obligations.  Contract  Closeout  or  Tennination  signifies  the  end  of  the  contractual 
obligation  between  buyer  and  seller.  It  also  involves  ensuring  that  all  administrative 
obligations  stemming  from  the  contract  are  complete  (Rendon  &  Snider,  2008). 

These  phases  logically  follow  the  lifecycle  of  a  contract  from  start  to  finish.  The 
SMCT  is  organized  in  a  similar  fashion.  While  not  specifically  aligned  with  the  six 
phases  of  the  contract  management  process  as  identified  by  Garrett,  the  subject  area 
codes  in  the  SMCT  follow  the  natural  progression  of  the  lifecycle  of  a  contract.  Subject 
area  code  1  is  contracting  customer  support  tasks;  subject  area  code  2  is  contracting  pre¬ 
award  tasks;  subject  area  code  3  is  contracting  post-award  tasks;  and  subject  area  code  4 
is  other  contracting  officer  tasks. 

F.  SMCT 

The  Soldier’s  Manual  of  Common  Tasks  identifies  training  requirements  for 
soldiers,  supervisors,  trainers,  and  commanders  in  career-management  field  5 1C.  It  is 
used  by  commanders  to  plan,  conduct,  and  evaluate  individual  training  (DoA,  2009, 
September  22).  The  Army  training  strategy  relies  on  soldiers,  noncommissioned  officers, 
and  officers  to  be  trained  and  ready  to  fight  our  nation’s  wars.  This  dictum  demands  that 
soldiers  and  their  commanders  continuously  plan  and  conduct  realistic  training.  The 
SMCT  provides  the  framework  of  that  training  for  CCOs.  The  36  training  tasks  that 
comprise  the  body  of  knowledge  for  a  CCO  are  broken  down  into  four  subject  area  codes. 
The  four  subject  codes  are  as  follows: 

1.  Contracting  Customer  Support  Tasks 

2.  Contracting  Pre-award  Tasks 

3.  Contracting  Post-award  Tasks 

4.  Other  Contracting  Officer  Tasks 

(DoA,  2009,  September  22) 

Each  subject  code,  in  turn,  has  corresponding  tasks.  Figure  11  depicts  the  36  SMCT 
tasks.  Subject  codes  are  not  in  any  particular  order  of  difficulty  or  skill.  All  CCOs, 
regardless  of  rank,  should  be  proficient  in  all  tasks,  regardless  of  subject  code. 
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Subject 

Area 

Task 

Number 

Task 

Subject 

Area 

Task 

Number 

Task 

1 

1-1 

Identify  Supported  Units 

3 

3-4 

Exercise  Options 

1 

1-2 

Train  Supported  Units  (Regarding  Contracting 

Proceedures) 

3 

3-5 

Monitor  Contract  Performance 

3 

3-6 

Process  Documents  For  Payment 

1 

1-3 

Advise  Supported  Units  (Regarding  Contracting  Matters) 

3 

3-7 

Modify  Contracts 

3 

3-8 

Terminate  Contracts 

1 

1-4 

Prepare  Contracting  Support  Integration  Plans 

3 

3-9 

Conduct  Contract  Closeout 

2 

2-1 

Review  Purchase  Requests  and  Supporting  Documents 

3 

3-10 

Process  Protests 

2 

2-2 

Conduct  Market  Research 

3 

3-11 

Process  Claims 

2 

2-3 

Conduct  Acquisition  Planning 

3 

3-12 

Process  Freedom  Of  Information  Act  Requests 

2 

2-4 

Document  Other  Than  Full  and  Open  Competition 

3 

3-13 

Process  Unauthorized  Commitment  s 

2 

2-5 

Train,  Designate,  Monitor  And  Terminate  Contracting 
Officer  Representatives  (CORs) 

4 

4-1 

Use  Standard  Form  44s  (SF  44) 

4 

4-2 

Train.  Designate.  Monitor  And  Terminate  a  Field 
Ordering  Officer  (FOO) 

2 

2-6 

Train,  Designate  Monitor  and  Terminate  Ordering  Officers 
(OO) 

2 

2-7 

Synopsize  Proposed  Contract  Actions 

4 

4-3 

Issue  Letter  Contracts 

2 

2-8 

Solicit  Competition 

4 

4-4 

Conduct  Vendor  Education 

2 

2-9 

Receive  Solicitation  Responses 

4 

4-5 

Employ  The  Government  Purchase  Card  (GPC)  As 
Payment  Method  and  Procurement  Tool 

2 

2-10 

Evaluate  Offers 

2 

2-11 

Prepare  Contract  Awards 

4 

4-6 

Manage  Bulk  Funds 

3 

3-1 

Conduct  Post  Award  Orientations 

4 

4-7 

Prepare  For  Deployment 

3 

3-2 

Notify  Unsuccessful  Offerors- Award  Synopsis 

4 

4-8 

Maintain  PIIN  Logs 

3 

3-3 

Issue  Task  Orders/Delivery  Orders  or  BPA  Calls 

Figure  11.  36  MOS  51  Common  Tasks  ( SMCT) 

(ECC,  2009,  September  22) 

Each  of  the  36  tasks  is  broken  down  into  conditions,  standards,  and  performance 
steps.  The  conditions  of  the  task  identify  “any  equipment,  tools,  references,  job  aids,  and 
supporting  personnel  the  Soldier  needs  to  use  to  perform  the  task  in  wartime.  It  also 
identifies  any  environmental  conditions  that  can  alter  task  performance  such  as  visibility, 
temperature  or  wind”  (DoA,  2009,  September  22).  The  standards  are  normally  described 
in  terms  of  accuracy,  completeness,  and  speed  and  serve  to  describe  how  well  and  to 
what  level  the  task  must  be  performed  during  wartime.  The  performance  steps  comprise 
the  training  and  evaluation  section.  These  steps  identify  specific  actions  that  a  Soldier 
must  do  to  successfully  complete  the  task.  In  terms  of  developing  the  PAT,  the 
conditions  will  help  develop  the  scenarios,  and  the  performance  steps  will  help  develop 
the  answers  to  the  test. 
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G.  SUMMARY 

This  chapter  described  in  detail  the  Expeditionary  Contracting  Command  and  the 
Soldier’s  Manual  of  Common  Tasks.  The  chapter  also  described  the  conditions  under 
which  the  ECC  was  established,  its  organizational  structure,  and  its  purpose.  Moreover, 
it  broke  down  the  subordinate  units  within  the  ECC,  from  Headquarters  to  individual 
CCO.  Next,  the  chapter  described  in  detail  the  multitude  of  training  requirements  for 
CCOs  within  the  framework  of  the  ECC’s  training  methodology.  Finally,  it  broke  down 
the  individual  subcomponents  of  the  SMCT. 

The  ECC  was  established  in  short  order  and  has  put  in  to  place  many  systems  to 
improve  the  numerous  shortfalls  identified  in  the  Gansler  Commission’s  report  and  other 
governmental  oversight  reports.  It  is  striving  to  train  and  develop  400  newly  assigned 
uniformed  CCOs  and  1,000  civilian  contracting  personnel.  In  doing  so,  it  must  also 
maintain  command  and  control  of  current  CCOs.  It  has  developed  a  model  training  plan 
for  CCOs  and  provided  the  training  doctrine  in  the  fonn  of  the  Soldier’s  Manual  of 
Common  Tasks  to  support  their  goals.  However,  the  SMCT  remains  incomplete  without  a 
formalized  proficiency-assessment  test  to  measure  a  CCO’s  perfonnance  against  the 
SMCT  standards.  The  purpose  of  this  research  is  to  develop  such  a  test.  Chapter  IV, 
PAT  Development,  will  cover  in  detail  the  process  the  research  team  used  to  develop  the 
PAT.  Chapter  IV  also  describes  in  detail  how  the  proficiency  assessment  test  was 
created.  It  describes  the  research  team’s  use  of  the  test-construction  theoretical 
framework  and  accounts  for  any  deviations  made  in  that  framework  due  to  time  or 
academic  constraints. 
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IV.  PAT  DEVELOPMENT 


A.  INTRODUCTION 

This  chapter  discusses  the  steps  and  methodology  the  research  team  took  in 
developing  the  PAT.  In  this  chapter,  we  explain  how  we  applied  the  theoretical  academic 
principles  of  test  construction  we  learned  during  our  literature  review  and  how  we 
applied  them  given  our  limitations,  both  as  students  (non-professional  test  writers)  and  in 
tenns  of  time  constraints,  given  our  academic  schedule. 

B.  TEST  CONSTRUCTION  PLAN 

The  development  of  the  PAT  was  planned  with  the  test-user  and  test-taker  in 
mind.  The  research  team  had  to  build  the  PAT  to  meet  the  needs  of  the  Expeditionary 
Contracting  Command  and  the  CCOs  who  would  eventually  take  the  test.  We  envisioned 
the  PAT  being  taken  in  a  digitized  fonnat,  as  this  would  be  the  most  efficient  manner  in 
both  administering  the  test,  and  in  taking  it.  Moreover,  a  digitized  version  of  the  test 
would  be  less  susceptible  to  compromise  by  would-be  test  takers,  and  would  lend  itself 
better  to  modifications.  While  placing  the  PAT  in  its  final  digitized  version  was  not  part 
of  the  scope  of  this  project,  the  questions  and  answers  written  by  the  research  team  had  to 
be  developed  in  a  manner  that  would  make  a  transfer  to  digitized  form  convenient. 

In  developing  the  PAT,  the  research  team  could  not  follow  the  steps  to  test 
construction  identified  in  the  literature  review  as  describe  by  either  Downing  &  Haladyna 
(2006)  or  Allen  &  Yen  (2001).  We  knew  early  on  that  our  academic  calendar  and 
limited  knowledge  in  test  construction  would  not  allow  us  to  develop  the  test  in  this 
manner.  Instead,  we  prioritized  the  major  steps  that  were  common  between  the  two  lists 
and  focused  on  test  blueprinting  and  item  writing. 

C.  TEST  BLUEPRINT 

The  research  team  first  developed  a  test  blueprint  for  each  of  the  tasks.  The  test 
blueprint  acted  as  a  road  map  for  test  construction  by  laying  out  which  subject  areas  we 
wanted  to  test  and  at  what  cognitive  domains  the  subject  area  should  be  tested.  This  test 
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blueprint  broke  down  each  of  the  tasks  into  subcomponents  called  items.  Each  item  is  a 
specific  subset  of  knowledge  of  that  particular  task.  The  research  team  developed  a 
spreadsheet  that  listed  each  of  the  items  within  a  specific  task  with  several  questions 
corresponding  to  each  of  item.  We  then  wrote  questions  that  tested  knowledge  of  that 
item.  As  we  wrote  questions  for  each  task,  we  annotated  on  the  spreadsheet  which 
cognitive  domain  level  the  questions  tested.  In  this  way,  we  were  able  to  determine  when 
we  had  adequate  numbers  of  questions  for  each  item  that  tested  at  each  cognitive  domain 
category. 

D.  ITEM  WRITING 

The  research  team  took  each  task  and  broke  it  down  to  its  subcomponents  of  task, 
conditions,  and  performance  steps.  The  tasks  in  the  SMCT  represent  the  36  testable  tasks 
that  we  used  to  measure  proficiency.  Having  the  task,  conditions  and  performance  test 
essentially  gave  us  the  questions  and  answers.  We  used  these  elements  of  the  tasks  of  the 
SMCT  to  construct  the  test  questions. 

The  research  team  used  the  knowledge  gained  during  the  literature  review  and 
planned  the  PAT  as  a  criterion-referenced  test.  Criterion-referenced  tests  attempt  to 
determine  if  an  individual  has  mastered  a  subject  area  in  a  specific  learning  objective 
(Marshall  &  Hales,  1972).  In  the  case  of  the  PAT,  this  test  would  attempt  to  determine  of 
CCOs  had  mastered  the  particular  task  within  the  SMCT. 

E.  VALIDITY  AND  RELIABILITY 

Test  validity  and  reliability  were  two  essential  test  characteristics  we  identified 
during  our  literature  review  that  we  had  to  pay  attention  to  during  item  writing.  The 
research  team,  as  much  as  possible,  wanted  to  ensure  the  PAT  was  written  with  these  two 
attributes  in  mind.  We  accepted  that  we  would  not  be  able  to  measure  either  validity  or 
reliability  in  our  test,  as  doing  so  would  involve  issuing  the  test  to  sample  populations 
and  measuring  consistency  in  responses  (Allen  &  Yen,  2001;  APA,  1974).  The  academic 
and  time  constraints  would  not  allow  us  to  follow  through  in  the  test  development  in  this 
manner.  However,  we  did  take  into  account  factors  to  maximize  both  validity  and 
reliability  in  our  test. 
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To  ensure  reliability,  the  research  team  incorporated  guidelines  for  item  writing  as 
presented  by  Kline  (1986).  These  guidelines  included  (1)  reducing  the  insight  which 
subjects  may  have  into  the  items,  (2)  writing  in  clear,  unambiguous  terms,  (3)  referring  to 
specific  instead  of  general  behavior,  (4)  ensuring  each  item  asked  only  one  question  or 
made  one  statement,  (5)  and  avoiding  as  much  as  possible  terms  of  frequency  (Kline, 
1986). 

Additionally,  we  looked  at  test  length  to  increase  reliability.  Generally,  the  longer 
the  test,  the  more  reliable  it  is  (Nunnally  &  Bernstein,  1978).  Based  on  this,  the  research 
team  attempted,  whenever  possible,  to  write  at  least  twenty  items  for  each  task.  The 
length  of  the  task,  conditions  and  perfonnance  measures  for  each  of  the  tasks  varied  and 
as  a  result,  so  did  the  number  of  items  for  each  task.  Some  tasks  required  well  over  thirty 
questions,  while  some  were  so  short  that  only  ten  could  be  developed. 

In  an  effort  to  prevent  compromise  of  the  test  by  would  be  test  takers,  the  research 
team  doubled  the  number  of  questions  for  each  task  from  an  average  goal  of  twenty 
questions  per  task  to  an  average  goal  of  forty  questions  per  task.  This  doubling  of  items 
would  serve  to  increase  the  database  of  questions  for  a  digitized  version  of  the  test,  thus 
allowing  for  multiple  versions  of  the  test  to  be  administered.  In  addition,  the  doubling  of 
items  would  result  in  the  greater  validity. 

The  research  team  also  took  in  to  account  efforts  in  the  development  of  the  PAT 
to  ensure  content  validity  of  the  test.  According  to  Nunnally  (1972),  validity  is  how  well 
a  test  meets  its  designed  function.  The  PAT  was  designed  to  measure  proficiency  in  the 
36  tasks  listed  in  the  SMCT.  One  of  the  ways  that  the  research  team  ensured  the  test 
measured  the  content  of  those  tasks  was  by  deconstructing  the  task,  conditions,  and 
performance  measures  in  the  SMCT  and  rewriting  them  as  questions.  For  example,  for 
Task  1-1,  Identify  Supported  Units,  one  of  the  performance  steps  is  to  recognize  key 
personnel  and  support  elements.  The  SMCT  lists  several  key  positions  the  CCO  may 
come  in  contact  with.  Under  the  position  of  Finance  Officer,  it  states,  “Key  element 
responsible  for  banking  and  currency  support,  disbursing  functions,  and  electronic  funds 
transfers.”  In  order  to  ensure  that  the  question  measured  the  content  of  the  performance 

step,  the  research  team  would  rephrase  the  verbiage  in  the  SMCT  in  the  form  of  a 
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question  as  such:  “Who  is  responsible  for  banking  and  currency  support,  disbursing 
functions,  and  electronic  funds  transfers?”  This  question  would  be  followed  by  several 
possible  answers,  one  of  which  would  be  the  correct  answer  (Finance  Officer).  The  other 
(incorrect)  answers  would  be  the  other  ones  from  the  list  of  the  other  key  personnel 
described  in  the  SMCT. 

Once  the  questions  for  each  subject  area  of  questions  were  completed,  they  were 
e-mailed,  in  a  Word  document,  to  our  sponsor  at  the  Expeditionary  Contracting 
Command.  Our  sponsor  would  then  mail  these  questions  out  to  senior  civilian  and 
military  CCOs  for  vetting.  This  vetting  process  by  senior  leaders  and  experts  in  the  field 
of  contingency  contracting  was  another  way  to  ensure  content  validity  (Nunnally  & 
Bernstein,  1978). 

F.  COGNITIVE  DOMAINS 

The  research  team  took  in  to  account  the  application  and  use  of  the  test  when 
determining  the  cognitive  domains  that  the  test  would  measure.  The  ECC  needed  the 
PAT  to  administer  to  thousands  of  CCOs  throughout  the  Anny.  This  necessitated  a  test 
that  could  be  administered  to  numerous  individuals,  at  different  times  and  locations,  by 
numerous  test  administrators.  Because  of  this,  the  final  version  of  the  test  would  need  to 
be  computer-based.  For  this  reason,  most  all  of  the  questions  and  answers  were  written 
in  a  multiple-choice  fonnat.  As  such,  most  of  the  questions  written  tested  knowledge  at 
the  remember  and  understand  categories,  and  only  to  a  limited  extent,  application 
categories  of  Bloom’s  cognitive  domains. 

As  previously  stated,  most  all  of  the  questions  were  written  as  what  are  commonly 
referred  to  as  multiple-choice  questions.  These  multiple-choice  questions,  known  as 
selected-response  items,  occur  when  the  examinee  is  given  both  the  problem  and  the 
correct  solution  and  then  must  choose  the  correct  solution  over  the  incorrect  solutions. 
Selected-response  items  were  chosen  over-constructive  response  type  questions  for  the 
PAT  since  selected-response  were  less  resource  intensive  and  more  conducive  to  a 
digitized  fonnat.  The  research  team  expanded  the  format  of  selected-response  questions 
to  include  true/false,  matching  and  sentence  completion. 
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Wood  (1960)  suggests  item  writers  should  establish  a  random  system  to  position 
the  correct  response  when  developing  selective-response  items  in  order  to  avoid  bias 
within  the  test.  The  reasoning  for  this  is  that  test  developers  will  inadvertently  choose  a 
particular  response  position  over  others  and  this  may  induce  bias  in  the  test.  In  other 
words,  a  test  developer  could  inadvertently  end  up  with  the  response  letter  A,  for 
example,  as  the  answer  to  most  of  the  multiple-choice  questions.  In  order  to  mitigate 
this,  every  question  the  research  team  wrote  had  the  first  response,  or  the  answer  letter  A, 
as  the  correct  answer.  In  this  way,  the  ECC  or  future  test  developers  who  transfer  the  test 
to  a  digitized  usable  form  could  randomly  shuffle  the  available  choices  and  the  correct 
response.  Additionally,  this  allowed  for  ease  of  vetting  by  the  ECC  for  validity  since  it 
eliminated  the  need  for  an  answer  key. 

G.  SUMMARY 

Over  the  course  of  four  months,  the  research  team  wrote  approximately  forty  test 
questions  for  each  of  the  36  tasks  in  the  SMCT.  The  tasks  varied  in  length  so  some  of  the 
tasks  had  as  few  as  10  questions  and  some  had  as  many  as  60.  In  total,  the  research  team 
wrote  over  1,100  questions  for  the  SMCT  spanning  over  300  pages  of  text.  Many  of  the 
questions  were  developed  to  test  redundant  items.  In  this  way,  these  questions  could  be 
used  in  different  versions  of  the  test.  The  question  and  answer  sheets  were  submitted  to 
our  sponsor  at  the  ECC  in  Word  format.  At  the  time  of  this  writing,  the  ECC  and  their 
senior  level  leaders  of  experienced  CCOs  had  received  all  questions  for  the  36  tasks  the 
team  developed.  The  next  chapter  is  the  conclusion  chapter  of  this  research  project.  It 
lists  recommendations  for  administration  of  the  test  and  suggests  areas  of  further 
research. 
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V.  SUMMARY,  CONCLUSION,  AND  AREAS  FOR  FURTHER 

RESEARCH 


A.  SUMMARY 

U.S.  Army  contingency  contracting  officers  (CCOs)  have  played  a  critical  role  in 
nearly  every  overseas  military  operation  by  providing  supplies  and  services  to  the 
warfighter  in  an  efficient,  timely,  and  ethical  manner.  Despite  this  critical  role,  the  Anny 
has  not  integrated  its  CCOs  or  the  procurement  process  in  a  holistic  approach  to  its 
expeditionary,  warfighting  mindset.  Shortfalls  in  the  number  of  U.S.  Army  CCO 
personnel  and  the  lack  of  a  standardized  training  process  have  negatively  affected  the 
Army’s  expeditionary  procurement  capabilities. 

The  U.S.  Army’s  Expeditionary  Contracting  Command  (ECC)  was  established  in 
January  2008  in  direct  response  to  the  Gansler  Commission’s  recommendations  to 
prepare,  train,  and  provide  skilled  contracting  personnel  in  support  of  U.S.  Army 
contingency  operations.  This  new  command,  and  other  resultant  changes  of  the  Gansler 
Commission’s  recommendations,  resulted  in  a  planned  net  increase  of  over  400 
uniformed  and  800  civilian  contracting  officer  positions.  The  ECC  developed  a 
standardized  set  of  36  tasks  to  properly  train  CCOs  prior  to  deployment.  These  36  tasks, 
published  in  the  ECC’s  Soldier’s  Manual  of  Common  Tasks  (SMCT)  MOS  5 1C 
Acquisition,  Logistics,  and  Technology,  represent  the  body  of  knowledge  that  a  5 1C 
CCO  must  know  to  be  successful  during  contingency  operations. 

The  establishment  and  standardization  of  training  tasks  by  the  ECC  for  CCOs  in 
the  MOS  5 1C  SMCT,  in  accordance  with  Army  training  doctrine,  has  helped  subordinate 
units  and  individual  CCOs  within  the  ECC  to  focus  their  training  efforts  prior  to 
deployment.  This  body  of  knowledge  is  a  significant  step  in  standardizing  the  pre¬ 
deployment  training  for  CCOs  and  preparing  them  for  their  mission  once  deployed.  To 
supplement  this  training,  an  appropriate  performance-assessment  tool  is  needed  to 
measure  performance  of  these  36  tasks.  Currently,  the  ECC  lacks  a  standardized  method 
of  measuring  individual  skill  and  task  proficiency  in  its  CCOs. 
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The  written  proficiency-assessment  test  questions  and  answers  developed  in  this 
research  will  allow  leaders  in  the  ECC  to  measure  a  CCO’s  knowledge,  skills,  and 
capabilities.  A  PAT  will  enable  leaders  to  measure  individual  task  proficiency  against 
established  standards,  determine  each  CCO’s  preparedness  to  conduct  duties  during 
deployment,  make  appropriate  CCO  assignments,  and  develop  individualized  training 
programs  for  CCOs. 

This  research  developed  the  questions  for  a  written  proficiency  test  based  on  the 
36  tasks  listed  in  the  SMCT.  The  research  team  developed  the  scenarios  and  questions  to 
assess  proficiency  in  accordance  with  conditions,  standards,  and  performance  steps  for 
each  of  the  36  tasks  in  the  SMCT.  Questions  were  structured  to  assess  Bloom’s  cognitive 
domains  of  remembering,  understanding,  and,  to  a  limited  extent,  application  of  tasks. 
The  research  team  vetted  each  of  the  scenarios  and  accompanying  questions  through 
experienced  CCO  focus  groups  within  the  ECC.  Upon  completion  of  the  test,  the 
research  team  provided  a  written  Contracting  Officer  Proficiency-Assessment  Test, 
encompassing  over  1,100  questions  covering  each  of  the  36  tasks  listed  in  the  SMCT,  to 
the  Expeditionary  Contracting  Command. 

B.  CONCLUSION 

The  results  of  this  research  project,  over  300  pages  containing  over  1,100  multiple 
choice  and  their  respective  answers,  was  developed  over  the  course  of  a  year.  These 
documents  represent  the  first  step  in  developing  a  proficiency  assessment  test  for  the 
Expeditionary  Contracting  Command’s  growing  ranks  of  CCOs. 

At  the  beginning  of  this  research  project,  we  posed  the  following  research 
question:  What  is  the  most  effective  and  efficient  method  of  measuring  Individual  CCO 
task  proficiency  in  the  36  tasks  listed  in  the  SMCT1  In  answering  this  question,  we  found 
that  a  multiple  choice  (selective-response)  type  test  administered  in  digitized  format 
directly  measuring  the  performance  measures  in  the  SMCT  would  be  the  answer  to  this 
question.  In  developing  the  PAT,  we  never  intended  to  convert  the  final  PAT  into  a 
digitized  version,  as  this  was  outside  the  scope  of  our  project.  We  did,  however,  write  the 
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test  questions  and  answers  that  would  go  into  such  a  test,  and  we  formatted  those 
questions  and  answers  in  such  a  way  that  could  easily  be  converted  into  a  digitized  test  by 
follow-on  researchers. 

Secondary  research  questions  we  asked  and  attempted  to  answer  were  as  follows: 

•  How  can  a  proficiency-assessment  test  be  developed?  In  answering  this 
question,  we  looked  to  core  test  principles,  test  construction  methods,  test 
blueprints  and  item  writing  covered  in  our  literature  review  for  the  answer. 
Instead  of  providing  the  answer  to  this  question,  we  used  the  knowledge 
gained  in  answering  the  question  to  develop  the  PAT  questions. 

•  What  type(s)  of  testing  is  best  suited  to  evaluate  the  cognitive  domains  of 
remembering,  understanding,  and,  to  a  limited  extent,  application  of 
knowledge?  As  in  the  previous  secondary  research  question,  the  answer  to 
this  question  helped  the  research  team  formulate  the  best  type  of  test  given 
the  requirements  of  the  ECC.  Given  that  any  final  version  of  the  PAT 
would  have  to  administered  at  various  locations  throughout  the  world  to 
numerous  CCOs  at  varying  times,  selective-response  were  deemed  best 
suited. 

•  What  type(s)  of  testing  is  most  efficient — in  terms  of  requiring  the  least 
amount  of  resources,  such  as  overhead  and  setup — for  the  tester  and  test 
taker?  This  last  question  identifies  the  need  for  the  PAT  to  be 
administered,  taken,  and  graded  in  a  digitized  format. 

The  answers  to  the  secondary  research  questions  were  discovered  by  conducting 
our  literature  review  on  test  construction  and  in  developing  the  test.  We  asked  and 
attempted  to  answer  these  questions  not  for  the  answers  in  and  of  themselves,  but  to  clue 
us  in  into  how  to  make  the  most  effective  proficiency-assessment  test  for  the  ECC.  In 
this  sense,  answering  these  research  questions  did  not,  by  themselves,  satisfy  the 
requirements  of  the  research  project.  Instead,  answering  these  research  questions  gave  us 
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the  know-how  in  how  to  write  the  PAT.  The  result  of  the  knowledge  gained  in  answering 
these  questions  is  the  over  1,100  questions  written  by  the  research  team  and  presented  to 
the  ECC  for  inclusion  the  PAT. 

At  the  conclusion  of  this  research  project,  all  question  and  answers  were 
submitted  to  the  ECC  for  review.  This  transferring  of  questions  and  answers  signifies 
transfer  of  ownership  of  this  body  of  knowledge  that  was  created  during  the  conduct  of 
this  research,  from  the  research  team,  to  the  Expeditionary  Contracting  Command.  The 
ECC  can  now  use  these  questions  to  complete  the  conversion  of  the  PAT  from  its  current 
raw  form  to  a  usable  and  implementable  test. 

C.  AREAS  FOR  FURTHER  RESEARCH 

There  are  several  areas  the  research  team  recommends  for  follow-on  research. 
This  research  was  able  to  develop  the  questions  and  answers  for  a  PAT  for  the  SMCT. 
However,  the  test  as  presented  in  this  research  project  is  not  in  a  usable  form.  Most  of 
our  recommendations  for  further  research  are  tied  to  the  continued  development  of  the 
PAT  as  a  usable  test  for  the  ECC.  Specifically,  the  research  team  makes  the  following 
recommendations:  (1)  Conversion  of  the  test  from  its  current,  written  form  to  a  digitized 
version,  (2)  Measure  the  test  for  validity  and  reliability,  (3)  Increase  the  number  of 
questions  in  the  question  and  answer  database,  (4)  Increase  the  number  of  scenario-  and 
performance-based  questions  that  measure  at  the  application  category  cognitive  domain 
and  (5)  Continually  validate  the  36  tasks  listed  in  the  SMCT  as  current  given  emerging 
trends  in  contingency-contracting. 

The  test,  in  its  present  form  cannot  be  administered.  The  product  produced  by 
this  research  project  resulted  in  over  1,100  questions  with  multiple  choice  answers,  the 
correct  answer  for  each  being  the  first  one,  or  A.  A  follow-on  group  of  researchers  could 
take  the  questions  and  answers  this  research  has  developed  and  convert  them  into 
digitized  version  that  could  be  administered,  taken,  and  graded  via  computer.  This 
computerized  test  could  serve  as  the  prototype  test  for  the  ECC  from  which  the  test  could 
be  measured  for  reliability  and  validity,  if  the  ECC  desired. 
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The  test  should  be  measured  for  validity  and  reliability.  While  we  took  steps  to 
mitigate  the  test  development  errors  that  affect  validity  and  reliability,  we  could  not  fully 
validate  the  test  as  valid  and  reliable.  A  follow  on  research  team  could  issue  the  test  in  a 
digitized  form  to  a  sample  pool  of  CCOs  and  measure  if  in  fact  the  test  is  an  accurate 
measure  of  the  knowledge  it  is  intended  to  measure  (validity)  and  that  it  continuously 
does  so  (reliability). 

The  test  should  include  additional  questions  in  multiple  combinations  to  create 
various  versions  of  the  test.  When  we  designed  the  test,  we  attempted  to  write  at  least 
twenty  questions  per  task.  In  an  effort  to  limit  test  memorization  and  compromise,  we 
increased  the  number  of  test  questions  to  forty  per  task  for  each  of  the  36  tasks.  These 
questions  may  have  to  be  replaced  during  the  vetting  process,  or  when  the  test  is 
measured  for  validity  and  or  reliability.  These  omitted  questions  should  be  replaced,  and 
new  questions  should  be  added  to  increase  the  number  of  questions  in  the  question 
database. 

Additionally,  questions  that  measure  more  demanding  cognitive  domain 
categories  such  as  application  of  knowledge  should  be  developed  for  the  test.  This  test, 
while  computer-administered  and  taken,  could  still  include  performance-based  questions 
where  test  takers  would  have  to  read  a  scenario  and  fill  out  proper  forms  related  to 
contingency-contracting.  This  research  effort  would  not  only  increase  the  quantity  of 
questions,  but  also  more  accurately  measure  a  CCO’s  abilities  to  perform  their  mission  by 
testing  at  a  more  demanding  cognitive  level. 

The  tasks  contained  in  the  SMCT  should  be  validated  against  emerging  trends  in 
contingency  contracting.  This  PAT  was  developed  based  on  the  36  tasks  in  the  SMCT. 
Our  literature  review  showed  that  tasks  in  contingency-contracting  training  changed  as 
the  nature  of  warfare  has  changed.  Based  on  this,  the  research  team  assumes  that  the  36 
tasks  in  the  SMCT  are  also  susceptible  obsolescence.  A  periodic  review  of  the  relevance 
of  the  36  tasks  contained  in  the  SMCT  would  assist  test  administrators  and  developers  of 
the  PAT  to  refine  both  the  SMCT  and  the  content  of  the  PAT. 
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Finally,  the  research  team  acknowledged  early  on  that  we  would  not  be  able  to 
fully  develop  the  PAT  into  its  final,  usable  form  as  a  computer  administered  test.  To  this 
end,  we  have  sought  to  establish  a  long-term  partnership  between  the  Naval  Postgraduate 
School  (NPS)  and  the  Expeditionary  Contracting  Command  to  continue  to  develop  and 
refine  the  proficiency-assessment  test  for  CCOs.  The  ECC  should  continue  to  reach  out 
to  the  NPS  to  facilitate  the  evolution  of  the  test.  Similarly,  NPS  faculty  and  students 
interested  in  the  field  of  study  of  contingency  contracting  should  look  to  ECC  for  real- 
world  problem  sets  with  which  to  assist. 

The  proficiency-assessment  test  as  presented  in  this  research  is  a  giant  step  in 
standardizing  an  individual  proficiency  assessment  tool.  While  it  is  not  immediately 
usable  and  implementable  by  the  ECC  in  its  current  form,  it  is  our  hope  that  the  products 
resulting  from  this  research  will  serve  the  ECC  as  solid  base  in  furtherance  of  this  effort. 
Once  implemented,  the  proficiency-assessment  test  submitted  as  part  of  this  research  will 
allow  leaders  in  the  ECC  to  measure  a  CCO’s  knowledge,  skills,  and  capabilities  by 
measuring  individual  task  proficiency  against  the  established  standards  contained  in  the 
SMCT.  The  PAT  will  finally  allow  leaders  within  the  ECC  to  effectively  and  efficiently 
determine  each  CCO’s  preparedness  to  conduct  duties  during  deployment,  make 
appropriate  CCO  assignments,  and  develop  individualized  training  programs  for  CCOs. 
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